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1.  INTRODUCTION  AND  SUMMARY 

At  the  third  meeting  of  the  Computer  Family  Architecture  (CFA)  Selection 
Cornfinttee  held  on  17-iiU  February  197fa,  it  was  agreed  that  final  selection  of 
the  CFA  would  be  based  not  only  on  the  architecture  features  that  had  been 
evaluated  oy  that  date  but  also  on  test  programs,  the  existing  software  bases, 
licensing  costs,  and  architecture  subsetabi 1 i ty/expandabi 1 i ty.  Subcommittees 
on  each  of  these  areas  wei^e  formed.  In  addition,  a selection  methodology  sub- 
conxnittee  was  forinea  and  asked  to  develop  a decision  procedure  to  integrate  the 
i results  of  tne  various  subcommittees. 

I Un  'c'J  February  197u,  the  Software  Evaluation  Methodology  Subcommittee  was 

t ■ requesteo  to  oevelop  the  criteria  for  evaluating  the  software  bases  of  the  CFA 

f finalists  and  tne  procedure  for  obtaining  a quantitative  evaluation.  This  docu- 

f ment  describes  this  i.iethoaology . Section  d of  this  volume  describes  the  problem 

I in  greater  detail.  Section  3 contains  the  technical  approach  which  includes  a 

. general  approach  to  the  structuring  of  a software  base.  Section  4 discusses  the 

f evaluation  criteria  utilized  to  rank  the  three  finalist  architectures  in  terms 

of  software  ease,  ana  Section  b contains  the  results  of  the  evaluation.  The 
remainder  of  this  section  is  a summary  of  the  methodology  utilized  to  evaluate 
tne  software  bases  ana  a summary  of  tne  results. 

Taole  I depicts  one  of  the  main  results  of  the  software  study.  It  shows  a 
comparison  of  tne  software  bases  and  deficiencies  for  the  three  finalist  archi- 
tectures. In  order  to  put  these  figures  in  context,  it  is  felt  that  a brief 
description  of  the  methodology  utilized  to  obtain  these  figures  (described  in 
later  sections)  is  necessary. 

The  rirst  part  of  tne  methoaology  was  to  compile  and  describe  all  tools  that 
could  in  any  way  be  utilized  for  the  development  of  DoO  software.  Next,  each 
voting  committee  i;ieinber  was  asked  to  vote  on  the  applicability  of  each  tool  as 
it  appliea  to  his/her  activity.  The  results  of  this  balloting  were  compiled 
and  tne  list  of  tools  was  generated  in  order  of  points  received.  Those  tools 
wnich  received  less  tnan  a threshola  of  l,uou  points  were  excluded  froin  further 
consiaeration  because  it  was  felt  that  it  would  not  pay  for  DoD  to  oevelop  tools 
with  such  a low  relative  score  in  applicability. 

The  next  step  in  the  process  was  to  have  the  manufacturers  determine  which 
tools  they  marketed.  This  information  was  audited  by  an  independent  agent  as 
to  its  accuracy.  Next,  software  tools  which  meet  the  tool  criteria  of  the  report 
referencea  aoove,  ano  are  marketed  by  other  than  the  architecture  manufacturers 
were  compiled,  both  of  these  lists  of  tools  were  combined  into  lists,  for  each 
architecture,  of  the  tools  available  and  not  available. 

In  oraer  to  convert  these  lists  into  dollar  figures,  an  estimate  of  the 
development  cost  of  each  tool  was  needed.  Each  manufacturer  was  requested  to 
estimate  the  size  of  each  tool  he  had  available  in  source  lines  of  code. 

Utilizing  productivity  figures  and  standard  costs  for  a man-year  of  effort,  a 
development  cost  for  each  tool  was  derived.  The  costs  were  compiled  for  each 
architecture,  to  determine  the  software  base  and  deficiencies  for  each  archi- 
tecture as  shown  in  Table  I. 
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Next,  it  was  deemed  necessary  to  estimate  the  number  of  years  it  would  re- 
quire the  government  to  make  each  architecture  totally  nondeficient  in  support 
software.  In  oroer  to  do  this,  the  three  lists  of  deficient  tools,  one  each  for 
each  architecture,  were  taken  and  oraered  in  terms  of  the  applicability  scored. 
Next,  these  lists  were  reoroered  slightly  in  terms  of  a PERT  sequence.  In  other 
woros,  if  tool  X scored  lower  in  applicability  than  tool  Y,  but  the  development 
of  X before  Y would  ease  the  development  of  Y or  other  tools,  then  tool  X would 
be  placed  higher  on  the  list. 

These  lists  were  tnen  converted  into  schedules  based  upon  anticipated  in- 
vestments of  5>1  million,  SJi  million  and  $3  million  a year  and  realistic  oevelop- 
ment  perioos.  From  these  nine  schedules.  Table  1 was  created  depicting  the 
years  it  woulo  take  to  bring  each  architecture  to  nondeficiency. 

This  oata  was  tlien  factored  into  two  distinct  cost  models  described  else- 
where to  obtain  best  estimates  of  the  life  cycle  costs  for  each  of  the  three 
archi tectures. 
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2.  STATEMENT  OF  THE  PROBLEM 

One  of  the  main  reasons  for  adopting  the  architecture  (i.e.,  instruction 
set)  of  an  existing  successful  computer  family  as  the  architecture  for  a future 
Military  Computer  Family  (MCF)  is  the  potential  utility  of  the  existing  software 
base.  This  was  supported  almost  unanimously  at  the  third  CFA  Selection  Committee 
meeting.  Use  of  the  term  "existing"  does  not  imply  that  MCF  interest  is  limited 
only  to  the  software  that  exists  on  23  August  197b  since  it  is  expected  that  each 
existing  software  base  will  continue  to  expand.  In  fact,  a premise  of  the  MCF 
program  is  that  if  the  CFA  selected  is  the  same  as  that  used  in  an  existing  and 
very  successful  computer  family,  that  CFA  will  continue  to  exist  and  be  successful 
and  its  software  base  will  continue  to  expand  independently  of  the  MCF  program. 
Therefore,  the  existence  of  a CFA  software  base  that  has  a higher  measure  of  im- 
mediate utility  will  be  taken  as  an  indication  of  the  potential  utility  of  the 
future  software  base  of  that  CFA. 

There  is  a significant  amount  of  software  associated  with  each  of  the  archi- 
tecture finalists.  However,  a mere  listing  of  such  software  would  not  be  adequate 
for  relative  ranking  of  the  finalists.  The  problem  then  is  to  (1)  define  what  is 
meant  oy  the  terms  "software  base",  (2)  determine  what  should  and  can  be  measured, 
and  (3)  develop  a methodology  for  assessment  of  the  value  of  a given  software  base 
that  facilitates  a relative  quantitative  ranking  of  the  four  CFA  finalists  by  23 
August  197b. 
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3.  technical  approach 
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a.  Domain  of  the  Software  base 

In  the  coiiimercial  world  there  is  a good  deal  of  applications  software  that 
is  transportable  across  applications.  Examples  are  payroll  systems,  inventory 
systems,  and  general  ledger  (account/billing)  systems  which  are  developed  and 
supported  as  market  items.  The  military  world  of  tactical  and  strategic  systems 
has  not  been  able  to  achieve  such  transportaoi 1 i ty , primarily  due  to  the  dis- 
parity among  applications,  t e highly  complex  time-critical  and  sensor-oriented 
functions  involved,  and  the  relative  immaturity  of  its  systems  (i.e.,  it  is  not 
easy  to  isolate  commonality  at  the  early  stages  of  evolution  of  a complex  require- 
ment, however,  after  the  development  of  three  or  four  similar  systems,  common 
functions  should  begin  to  emerge). 

After  a orief  attempt  to  find  commonality  among  existing  military  applica- 
tions software  failed,  it  was  the  consensus  of  the  subcommittee  that  applications 
software  should  not  oe  considered  within  the  domain  of  the  software  base.  Only 
support  software  will  be  included.  Support  software  includes  that  software  used 
tor  producing,  modifying,  analyzing,  and  testing  a computer-based  system.  There 
will  oe  no  distinction  maoe  between  the  support  software  that  runs  in  the  opera- 
tional target  environment  (e.g.,  executive  systems  and  data-base  management  sys- 
tems) and  the  support  software  that  only  runs  in  the  development  (support/host) 
environment  (e.g.,  compilers  and  simulation  systems). 

The  section  describing  the  software  base  will  serve  as  a more  precise  oefini-  j 

tion  of  the  term  support  software.  Specifically,  any  software  defined  there  will  , 

oe  considered  to  be  support  software.  ] 

1 

b.  The  Military  Software  Development  Environment  I 

The  evaluation  of  support  software  cannot  be  completed  without  some  knowledge 
of  the  way  in  which  such  software  will  be  used  in  the  development  of  military  com- 
puter-oaseo  systems.  While  it  is  not  the  intent  of  this  report  to  aelineate  a 
complete  approach  to  the  software  development  environment  for  the  MCF,  the  report 
cannot  be  completed  without  a preliminary  consideration  of  this  important  area. 

For  example,  if  it  were  decreed  by  DOD  that  all  MCF  software  had  to  be  developed  | 

on  a CDC-b6UU  host  environment  and  then  transferred  to  the  MCF  target  hardware,  j 

then  it  would  probably  be  the  case  that  the  support  software  base  would  be  a non-  j 

issue  in  the  selection  of  the  CFA;  i.e.,  the  importance  of  the  C0C-fa6ULi  support  | 

software  base  would  far  outweigh  the  importance  of  the  software  bases  of  the  CFA  ; 

finalists,  and  the  CDC-bbUO  software  base  would  apply  equally  as  well  to  each  CFA  | 

finalist.  i 

i 

(1)  Host-Target  Concept  ^ 

(a)  General  < 

In  the  past,  far  too  many  small  to  medium  scale  military  computer  systems  i 

were  developed  using  the  computer  that  is  to  go  into  the  final  operational  sys-  j 

tern  as  its  own  software  development  tool.  Many  of  the  consequences  of  this  \ 

approach  were  disastrous  since  these  development  environments  are  virtually  de-  ; 

void  of  the  very  broad  spectrum  of  powerful  support  software  tools  that  now  exist  1 

on  larger  computers.  (Such  tools  will  be  described  in  detail  in  the  remaining  i 
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sections  of  this  report).  This  section  describes  an  approach  to  software  de- 
velopment in  which  programs  are  developed  on  one  computer,  the  host,  for  opera- 
tion on  another  computer,  the  target.  The  basic  rationale  for  this  approach  is 
to  optimize  the  host  computer  configuration  and  its  support  software  for  the  de- 
velopment task  and  to  allow  the  target  computer  configuration  to  be  optimized 
for  the  objectives  of  the  system  being  developed.  Such  an  approach  is  especially 
applicable  to  the  aevelopment  of  software  systems  for  military  tactical  appli- 
cations whose  computer  configurations  are  inade^juate  for  effective  use  of  high 
level  language  compiUrs  and  other  relatively  large  and  powerful  support  soft- 
ware tools. 

The  host-target  approach  lias  the  potential  of  minimizing  the  cost  of  support 
software  because  many  support  software  tools  are  independent  of  the  target  com- 
puter and  also  has  the  potential  of  reducing  development  costs  of  an  environment 
optimized  for  the  development  task. 

however,  the  host-target  approach  introduces  a significant  problem  in  the 
testing  activities  of  development  if  the  host  and  target  computers  are  of  dif- 
ferent architecture  families.  It  is  desirable  to  conduct  program  tests  on  the 
host  computer  in  order  to  miniiiiize  the  complexity  of  the  development  process 
and  to  benefit  from  the  test  support  software  of  the  host  system,  but  differences 
between  the  host  ano  target  architectures  may  affect  the  validity  of  such  testing. 

The  following  sections  describe  target-independent  development  activities, 
target-dependent  development  activities,  development  tools  affected  by  the  host/ 
target  approach,  and  the  implications  for  the  MCF. 

(b)  Target-Independent  Development  Activities 

Many  of  the  activities  of  a software  development  project  are,  at  least,  par- 
tially independent  of  the  target  computer  architecture.  This  includes  activities 
such  as  configuration  management  where  the  activity  itself  is  relatively  indepen- 
dent as  well  as  activities  such  as  editing  a source  program  where,  though  the 
activity  product  may  be  target-dependent,  the  support  software  tools  used  can  be 
independent  of  the  target  computer.  Some  examples  of  target-independent  activities 
are; 

(1)  Planning 

(2)  Requirement  Determination 

(3)  System  Design 

(4)  Design  Validation 

(5)  Program  Library  Management 

(o)  Documentation 

(7)  Configuration  Management 

Many  of  these  activities  are  now  being  performed  with  the  aid  of  computer- 
based  support  tools.  The  possibility  of  improving  the  development  process  by  use 
of  such  tools  has  been  recognized  and  there  is  a trend  toward  even  greater  use  of 
computers  to  support  these  activities.  The  culmination  of  this  trend  seems  like1\ 
to  be  the  totally  integrated  software  development  system,  a relatively  large,  inte- 
grated data  base  system  containing  a complete  description  of  the  development  project 
including  (1)  requirement  specifications,  (2)  component  representations  (design, 
program  code,  test  data,  documentation,  etc.)  (3)  information  about  the  project 
activities,  and  (4)  relations  among  the  requirements,  component  representations. 
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ana  activities.  Support  software  tools  will  oe  associated  with  the  data  base  to 
perform  generation,  analysis,  transformation,  and  reporting  of  the  project  descrip- 
tive aata.  A key  design  goal  for  the  support  system  is  to  automate  maintenance  of 
the  completeness,  currency,  ana  integrity  of  the  project  descriptive  data.  This 
will  have  a significant  positive  effect  on  the  visibility  of  project  progress, 
product  reliability,  and  maintenance  costs.  Totally  integrated  software  oeveop- 
ment  systems  are  now  being  developed.  Some  are  at  least  partly  operational  ana 
are  being  refined  and  extended.  It  is  clear  that  such  support  tods  are  expensive 
to  develop  ana  require  large-scale  computer  system:  (with  ‘Extensive  memory  ana  I/O  . 
capabilities)  for  effective  operation.  These  requirements  make  it  highly  unlikely 
that  such  software  support  systems  will  ever  be  developed  for  operation  on  the 
smaller  military  tactical  computers  or  their  commercial  counterparts.  (Even  the 
more  traaitional  large-scale  software  development  systems  cannot  be  supported  on 
such  small  computers.) 

Tne  potential  benefits  of  using  sjch  an  integrated  support  software  system 
and  the  high  cost  of  developing  it  corstitute  strong  arguments  for  centralizing 
software  development  support  on  one  or  a few  types  ^f  host  computer  even  though 
the  software  is  being  developed  for  operation  on  one  of  a variety  of  target 
computers. 

(c)  Target-Dependent  Development  Activities 

Several  activities  of  a software  development  project  are  dependent  on  the 
target  computer  in  which  the  aeveloped  programs  will  operate.  A host-target  ap- 
proach must  provide  for  this  dependency.  The  most  significant  of  these  aependent 
activities  is  testing.  Program  translation  also  has  some  aependent  aspects. 

Final  software  testing  and  some  categories  of  system  testing  require  opera- 
tion in  an  environment  as  close  to  the  application  as  possible.  A host-target 
approach  does  not  provide  a rationale  fo*^  relaxing  this  requirement;  therefore, 
it  is  necessary  to  provide  for  some  target  computer  use  during  development.  One 
of  the  objectives  of  the  testing  of  individual  program  components  ana  groups  of 
related  programs  is  to  provide  a oasis  for  confidence  that  the  system  ana  final 
software  tests  can  be  completed  without  extensive  changes.  Therefore  it  is  bene- 
ficial for  the  environment  of  program  testing  to  closely  resemble  the  operational 
environment  also,  although  this  requirement  is  not  as  demanding  as  tlie  system 
test  requi reiiient.  If  tfe  host  computer  architecture  differs  from  that  of  the 
target  computer,  then  the  amount  of  final  software  testing  that  is  possible  on 
the  host  computer  will  be  limited  to  that  which  can  be  done  through  simulation 
(and  possibly  emulation)  of  the  target  computer.  Simulation/emulation  may  be  a 
reasonably  effective  substitute  for  the  target  system  in  a good  portion  of  the 
final  software  testing  if  the  target  is  a small-scale  computer  system.  However, 
if  the  target  system  is,  e.g.,  a command  and  control  system  that  employs  a large 
scale  computer  system  with  extensive  I/O  operations  and  if  the  host  and  target 
architectures  are  incompatible,  then  software/system  testing  will  be  severely 
hampered  because  the  host  will  waste  too  much  time  in  its  mimicking  of  the  target 
architecture  to  be  an  effective  test  vehicle. 

Additional  target  dependent  activities  are  compilation,  assembly,  and 
link-editing.  However,  these  dependencies  are  significantly  different  from  the 
testing  dependency.  They  result  primarily  from  the  fact  that  most  existing  pro- 
gram translators  have  been  built  to  operate  on  and  produce  code  for  the  same 


8 


computer  architecture.  This  dependency  does  not  result  from  the  intrinsic  re- 


quirements of  the  translation  process.  Basically,  program  translation  is  a data 
processing  operation  which  is  relatively  independent  of  the  computer  on  which  it 
is  performed  and  which  can  be  most  efficiently  performed  in  the  relatively  large 
memory  spaces  of  host  computers. 

(d)  Host-Target  Development  Tools 

This  section  discusses  the  impact  of  the  host-target  approach  on  three 
classes  of  support  tools: 

(1)  Compilers  and  Assemblers 
(Z)  Target  Simulators  or  Emulators 
(3)  Host-Target  Communication 

One  possible  approach  to  achieving  the  transfer  from  test  operation  on  the 
host  computer  to  operation  on  the  target  computer  is  to  program  in  a high  level 
language  and  to  use  two  compilers,  one  producing  host  computer  code  and  the  other 
producing  target  computer  code.  However  this  approach  is  not  generally  applicable 
within  the  current  state-of-the-art  because  most  currently  available  programming 
languages  are  not  sufficiently  standardized  and  machine-independent  for  trouble- 
free  transfer  of  programs  from  one  machine  architecture  to  another  and  because 
the  requirements  for  many  military  computer-base  systems  cannot  be  met  without 
machine-dependent  programming.  Where  requirements  permit,  this  approach  should 
be  considered  but  it  should  not  be  expected  to  be  uniformly  applicable. 

Another  approach  to  the  problem  of  transfer  from  host  to  target  computer  is 
to  use  cross  translators  and  a simulator  or  emulator  of  the  target  computer  opera- 
ting on  the  host  computer.  The  cross  translator  operates  on  the  host  computer 
and  translates  source  programs  to  target  machine  object  programs.  Target  com- 
puter programs  may  be  tested  by  interpretation  on  the  host  computer  through  use 
of  a target  computer  simulator  program  or  emulator  microprogram.  It  is  possible 
to  faithfully  imitate  the  logical  characteristics  of  the  target  computer  by  simu- 
lation or  enwlation  but  usually  it  is  not  feasible  to  match  the  target  computer's 
timing  characteristics.  While  emulation  of  the  target  computer  by  microprogramming 
the  host  computer  may  provide  a closer  approximation  of  the  target  computer's  tim- 
ing characteristics  as  well  as  significantly  greater  throughput  than  simulation, 
the  number  of  available  computer  models  suitable  for  emulation  of  a variety  of  tar- 
get computers  is  quite  limited  at  the  present  time.  Since  the  host  system  will 
most  likely  be  commercial  hardware  (due  to  cost  and  availability  factors),  the 
general  purpose  emulation  capability  cannot  be  counted  on. 

Testing  conducted  on  the  target  computer  may  also  be  supported  by  tools  opera- 
ting on  the  host  computer.  Some  of  the  possible  types  of  such  support  are: 

(1)  Transmission  of  Programs  from  Host  to  Target 
(Z)  Storage  of  Test  Data  Recordings 

(3)  Test  Data  Analysis  and  Reduction 

(4)  Simulation  of  the  Application  Environment 
(b)  Monitoring  and  Controlling  Test  Operation 

One  0^  the  most  critical  basic  test  support  capabilities  is  automated 
communication  between' the  host  and  target  computers  for  transmission  of  pro- 
grams, data,  and  test  control  commands.  If  the  computer  hardware  is  suitable. 
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the  most  effective  test  interface  woula  permit  the  host  computer  to  oirectiy 
address  the  target  computer  memory  and  to  interrupt  its  CPU. 

(e)  Summary  of  Host-Target  Concept 


Applying  a host-target  approach  to  the  software  development- process  in- 
; volves  ooth  advantages  and  disadvantages.  The  advantages  proceed  from  the 

possibility  of  using  a standardized,  comprehensive,  and  coordinateo  set  of 
support  software  tools.  The  disadvantages  proceed  from  the  acdeo  development 
task  of  transferring  the  development  programs  from  the  host  to  the  target  com- 
puter. Ihe  advantage  of  using  a host  computer  optimized  for  software  develop- 
ment while  allowing  the  target  computer  to  be  optimized  for  its  particular 
application  is  significant.  However,  if  the  host  and  target  computers  are  of 
different  architecture  families,  then  the  disadvantages  become  severe,  especially 
with  regard  to  the  use  of  the  host  in  the  development  cf  the  larger  military 
computer  systems  such  as  cotranand  and  control  systems.  On  the  other  hano,  if  the 
host  and  target  computers  use  the  same  architecture,  and  if  the  commercial  host 
has  a powerful  software  development  system,  then  a major  portion  of  the  soft- 
; ware  development  and  testing  may  be  accomplished  on  the  host  ana  the  programs 

may  be  transferred  to  the  target  without  difficulty.  Therefore  it  is  concluaeo 
that  the  tactical  software  development  process  may  be  greatly  enhanced  through 
the  implementation  of  a host-target  concept  where  both  host  and  target  have  the 
same  architecture.  This  has  significant  implications  with  respect  to  the  final 
choice  of  architecture  for  the  MCF. 

(2)  Software  Development  Activities 

In  this  section  a model  will  be  presented  that  aided  in  the  quantitative 
assessment  and  relative  ranking  of  the  CFA  finalists  with  respect  to  their  soft- 
ware bases.  Committee  members  were  asked  to  indicate  the  relative  importance 
of  the  various  items  (software  tools)  in  the  software  base  model  by  the  assign- 
ment of  weights  to  software  tool  types.  In  order  to  assign  weights  in  a meaning- 
ful way,  it  was  important  to  understand  not  only  what  the  tool  does  but  also  in 
which  part  of  the  software  development  cycle  it  is  used,  i.e.,  which  activity  it 
supports.  In  this  way,  a committee  member  was  able  to  place  more  weight  on  a 
tool  that  aids  an  activity  in  the  software  development  cycle  that  he  considered 
to  be  more  important.  To  aid  this  process,  each  tool  description  contained  a 
reference  to  the  applicable  activity  of  the  software  development  cycle.  In  this 
section,  the  software  development  cycle  activities  will  be  delineated. 

The  software  development  process  is  partitioned  into  the  following  major 
activities: 

(1)  Analyze  Requirements 

(2)  Design  Software 

(3)  Build  System  Tests 

(4)  Build  and  Unit-Test  Software 

(5)  Integrate  and  System  Test 

(6)  Maintain  System 

Figure  1 illustrates  these  software  development  activiti^rs  and  their  inter- 
relationships. The  lines  with  arrows  and  dots  at  each  end  indicate  both  an 
output  (result)  ana  feedback  path.  The  "tag"  on  each  such  line  separates  each 
action  by  a slash,  i.e.,  output/feedback.  Now,  each  activity  will  be  described. 
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In  3.3,  the  tools  that  support  a single  activity  as  well  as  those  tools  that  are 
common  across  activities  will  be  described  briefly. 

(a)  Analyze  Requirements  (Activity  1) 

"Analyze  Requirements"  performs  the  decomposition  of  the  user  needs  into 
the  functions  of  the  required  system.  Following  decomposition  and  the  develop- 
ment of  a functional  model,  functions  are  allocated  to  hardware,  software,  firm- 
ware, and  people.  Then  the  results  of  the  functional  decomposition,  allocation 
and  the  required  system  performance  parameters  are  used  to  search  a descriptive 
catalog  of  existing  systems  to  locate  suitable  candidates  for  reuse  or  modifica- 
tion. The  systems  (if  any)  resulting  from  this  search  and  any  new  functions  that 
must  be  developed  may  be  simulated  to  determine  their  gross  performance  character- 
istics. This  activity  is  controlled  by  the  analysts'  knowledge  of  the  current 
state  of  the  art,  and  the  available  budget  for  the  proposed  new  system.  If  a de- 
cision is  taken  to  proceed  with  development,  a software  functional  design  specifi- 
cation is  produced  and  recorded,  and  is  used  to  control  the  "Design  Software" 
activities.  A project  schedule  is  also  produced. 

(b)  Design  Software  (Activity  2) 

"Design  Software"  uses  the  functional  design  specification  allocated  to 
software  to  produce  the  implementation  specification.  It  has  available  a library 
of  "proven  algorithms"  to  assist  in  design,  and  has  the  ability  to  respond  to  the 
"Analyze  Requirements"  activity  with  a "can't  design"  or  "can't  meet  schedule." 

The  output  of  this  activity  is  the  implementation  specifications  that  will  con- 
trol the  software  unit  build  and  integration  activities  (Activities  4 and  5). 

(c)  BuilG  System  Tests  (Activity  3) 

"builo  System  Tests"  is  the  activity  that  designs  and  constructs  the  sys- 
tem acceptance  tests.  Note  that  it  is  controlled  by  the  same  set  of  functional 
specifications  that  control  the  "Design  Software"  activity,  and  that  it  is  un- 
constrained by  and,  therefore,  may  proceed  in  parallel  with  "Design  Software" 
ana  "Build  and  Unit  Test  Software."  The  activity  has  a library  of  previously 
constructed  tests  that  are  presuiiiably  tied  to  subsystems  that  will  be  reused  as 
directea  by  Activity  1.  The  output  of  this  activity  is  the  set  of  system  test 
scenarios,  drivers  and  monitors  that  will  control  Activity  5,  "Integrate  and 
System  Test. " 

(d)  Build  and  Unit-Test  Software  (Activity  4) 

"Build  and  Unit  Test  Software"  uses  the  implementation  specifications  pro- 
duced by  Activity  2 to  produce  unit  tested  modules  of  software.  It  has  avail- 
able a library  previously  constructed  of  modules  that  can  be  reused  with  or 
without  modification. 

(e)  Integrate  and  System  Test  (Activity  b) 

"Integrate  and  System  Test"  uses  the  modules  produced  by  Activity  4 and 
the  interface  and  subsystem  specifications  produced  by  Activity  3 to  bind  the 
system  components  into  their  final  form.  It  then  exercises  the  system  using 
the  test  scenarios  and  monitors  provided  by  Activity  3 to  validate  the  system. 

Its  final  output  is  the  completed  system  released  to  the  maintenance,  distri- 
bution and  configuration  control  activity  (Activity  6).  Integration  and/or 
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test  failures  are  reflected  back  to  the  appropriate  design,  test  production  or 
software  production  activities. 

(f^  Maintain  System  Activity  (Activity  6) 

The  final  oevelopment  activity,  "Maintain  System"  is  primarily  a clearing- 
house and  control  center  for  the  reception,  evaluation  and  control  of  engineering 
change  requests.  These  are  routed  to  the  appropriate  activity  for  re-implemen- 
tation, re-design,  o-  re-analysis.  The  maintenance  function  distributes  con- 
figuration-controlled systems  to  the  users,  and  releases  the  results  of  the 
development  effort  into  the  available  technology  data-base. 

c.  Structuring  of  the  Software  Base 
(1)  General 

The  software  base  will  be  structured,  in  part,  by  partitioning  the  software 
tool  types  according  to  the  specific  development  activities  they  support.  How- 
ever, this  approach  may  conceal  that  part  of  the  software  base  that  supports  the 
operation  of  such  tools  and  does  not  clearly  indicate  those  tools  that  support 
all  activities.  To  make  such  software  visible,  the  software  base  will  be  struc- 
tured further  through  a layered  approach  that  will  provide  insight  into  the  re- 
lationships among  software  base  components.  Further  structuring  of  the  software 
base  according  to  sub-activities  supported,  disciplines  supported--e.g. , manage- 
ment, engineering  (design  and  program  development),  testing,  simulation,  and  docu- 
mentation--or  according  to  evaluation  categories--e.g. , functionally,  reliability, 
performance,  cost,  and  mai ntai nabi 1 i ty--was  considered  but  deemed  impractical. 

There  are  at  least  five  distinct  layers  associated  with  an  operational  com- 
puter system  (Figure  2)  and  three  layers  of  software  that  support  the  software 
development  process. 

Layer  U represents  the  bare  computer  hardware  including  items  such  as 
processors,  channels,  main  storage,  mass  storage,  bulk  I/O,  archival  storage, 
hardware  monitors,  terminals,  sensors,  and  communications  interface  devices. 

Layers  1 through  3 "reside"  on  the  hardware  and  collectively  provide  the  virtual 
machine  capability  that  is  necessary  to  support  layer  4 the  development  of  ap- 
plications software.  In  the  following  paragraphs,  layers  1 through  3 are  des- 
cribed along  with  short  descriptions  of  the  tools  that  reside  in  these  layers 
and  the  relationship  of  such  tools  to  the  various  development  activities  dis- 
cussed in  3.2.2. 

For  more  detailed  descriptions,  the  reader  is  referred  to  Appendix  A. 

(2)  Layer  3;  Functional  Support  Tools 

Layer  3 contains  those  tools  that  provide  direct  support  to  the  six  soft- 
ware development  activities  of  Figure  1 and  are  the  tools  with  which  the  appli- 
cations software  developer  has  the  greatest  interaction.  Layer  3 tools  will  be 
related  to  the  specific  development  activities  they  support. 

(a)  Layer  3 Tool  Types  That  Support  Activity  1 (Analyze  Requirements) 

The  types  of  tools  that  are  directly  applicable  to  requirement  analysis 
are  listed  below: 
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SOFTWARE  BASE  STRUCTURE 


(1)  General  Purpose  System  Simulators 

Allows  a user  to  construct  a computer  model  of  a real 
or  proposed  system  and  to  perform  simulation  experiments 
to  determine  the  behavior  of  the  tnooel  under  various 
operational  conditions. 

(li)  System  Description  Languages  i Analyzers 

Assist  system  analysts  in  describing  the  functional 
characteristics  of  a system  and  in  validating  the 
consistency  and  completeness  of  a functional 
decomposition. 

(b)  Layer  3 Tool  Types  That  Support  Activity  2 (Design  Software) 

The  types  of  tools  that  are  directly  applicable  to  the  software  design  are 
listed  below: 

(1)  Computer  System  Simulators 

Similar  in  nature  to  the  general  purpose  simulator 
except  that  its  basic  building  blocks  represent  real 
computer  system  components  whose  modeled  behavior 
approximates  the  throughputs,  capacities  and  access 
times  achievable  on  the  modeled  equipments. 

(2)  Data  Base  Design  Aids 

Assist  data  base  designers  in  grouping  data  elements 
into  logical  record  classes  and  in  determining  the 
relationships  among  logical  record  classes  implicit 
in  either  the  nature  of  the  data  or  the  usage  of  the 
data. 

(3)  Data  Dictionary  Systems 

Assist  data  base  designers  in  managing  the  data 
definition  activities. 

(c)  Layer  3 Tool  Types  That  Support  Activity  3 (Build  System  Tests) 

The  types  of  tools  required  to  support  system  test  construction  are  listed 
below: 

(1)  Test  Data  Generators 

Create  data  files  for  testing  and  validating 
computer  programs. 

(2)  Test  Data  Auditors 

Compare  data  files  against  specifications  and 
produce  reports  of  discrepancies  and/or 
compliance. 
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(3)  Test  Case  Design  Advisors 

Analyze  programs  written  in  a high  level 
language  and  present  the  results  of  that 
analysis  in  a form  suitable  to  assist  test 
case  designers  in  the  selection  of  test  data. 

(4)  Test  Instrumenters  and  Analyzers 

Instrument  modules  under  test  so  as  to  col- 
lect data  characterizing  the  behavior  of  the 
module. 

(d)  Layer  3 Tool  Types  That  Support  Activity  4 (Build  and  Unit  - Test 
Software) 

The  types  of  tools  that  are  required  support  the  program  development  and 
unit-test  activity  are  listed  below; 

(1)  Assemblers 

Allow  programs  to  be  coded  in  a symbolic 
language  in  which  statements  generally  cor- 
respond to  a single  machine  instruction. 

Cl)  Compilers 

Translate  programs  written  in  a high  level 
language  into  either  relocatable  object  code 
acceptable  to  a linker  or  assembly  language 
acceptable  to  an  assembler. 

(3)  Linkers 

Combine  the  text  produced  by  separate  invocations 
of  compilers  and  assemblers  ("object  modules") 
into  executable  code  strings  ("load  modules"  or 
"core  images")  that  can  be  loaded  into  the  com- 
puter's main  storage  and  executed  without  further 
pre-processing. 

(4)  Debugging  Aids 

Assist  the  programmer  in  locating  the  sources 
of  program  errors  that  have  been  discovered 
during  unit  testing,  usually  by  giving  him 
some  control  over  the  execution  of  the  module 
under  test  that  is  external  to  the  normal  pro- 
gram code. 

(5)  Module  Libraries  & Change  Control  Systems 

Provide  computer  controlled  maintenance  of 
groups  of  related  source  modules  (programs). 
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object  modules  (the  output  of  assemblers  and 
compilers),  and  load  modules  (the  output  of 
1 i nkers) . 

(6)  Performance  Monitors 

Assist  the  programmer  in  quantifying  the  resource 
consumption  characteristics  of  a program  and  in 
isolating  performance-critical  areas. 

(7)  Standards  Enforcers 

Allow  source  programs  to  be  examined  automatically 
and  checked  for  conformance  to  installation-de- 
fined standards  of  format,  content,  and  usage. 

(tt)  Preprocessors/Reformatters 

Assist  programmers  in  producing  well -structured 
and  readable  programs  by  allowing  the  introduc- 
tion of  structured  programming  constructs  into 
source  programs  for  languages  that  do  not  have 
them,  and  by  automatically  controlling  indentation, 
the  placement  of  comments,  etc.,  to  produce  read- 
able listings. 

(e)  Layer  3 Tool  Types  That  Support  Activity  5 (Integrate  and  System  Test) 

and  Activity  6 (Maintain  System  ) 

There  are  no  unique  layer  3 tools  that  exist  to  support  these  activities. 
The  tools  that  were  listed  for  activities  1 through  A are  generally  applicable 
Ln  activities  5 and  6 at  layer  3.  Most  of  the  tools  used  in  practice  that  are 
specifically  oriented  to  activity  S are  special-purpose,  e.g.,  test  environment 
tools  (emulators,  hot  benches,  system  integration  lab  support,  virtual  machines 
test  drivers  (drive  test  scenario  against  system;  use  test  monitor  to  collect, 
record,  and  evaluate  test  results)  and  special  performance  monitors. 

(3)  Layer  General  Support  Services 

The  primary  function  of  layer  2 tools  is  to  provide  a framework  of  comnion 
services  that  will  allow  the  outputs  of  third  layer  functions  to  be  stored, 
retrieved  and  inter-communicated.  Second  layer  functions  should  be  usable 
for  common  purposes  across  different  third  layer  functions,  and  should  serve 
to  hide  (where  possible)  differences  between  first  layer  and  third  layer  func- 
tions. Layer  2 tool  types  provide  general  support  to  all  of  the  software 
development  activities.  The  layer  2 tool  types  are  summarized  as  follows: 

(1)  Data  Base  Management  System 

Allow  the  user  of  a computer  system  to  define  the  contents 
of  and  the  logical  relationships  between  collections  of 
data  items  that  represent  some  useful  abstraction  of  a 
real-world  phenomenon  (tactical  command  and  control  system. 
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the  modules  and  documentation  of  a system  of  computer  pro- 
grams) without  being  concerned  with  the  physical  mechanics 
of  storing,  locating,  and  retrieving  items  or  groups  of 
i terns . 

{2)  HERT/CPM  Systems 

Assist  managers  in  planning  and  controlling  project  activities. 

(j)  Project  Estimation  Systems 

Assist  in  the  development  of  work  oreakdown  structures 
and  related  performance  standards  for  use  in  estimating 
project  resource  requirements. 

(4)  Documentation  Aids 

Assist  in  the  preparation  and  maintenance  of  documenta- 
tion about  the  modules  of  a system.  Aids  most  relevant 
to  a program  development  environment  include  text  proces- 
sing systems,  flowchart  construction  languages  and 
automatic  flowcharters. 

(5)  Data  Manipulation  Utilities 

Allow  the  system  user  to  alter  the  format  and  content 
of  data  files  independent  of  the  logical  significance 
of  the  data  fields  involved.  While  this  category  in- 
cludes the  standard  media  conversion  utilities  such 
as  card-to-tape,  tape-to-printer,  etc.,  these  have 
not  been  included  because  it  is  felt  that  simple  pro- 
grams of  this  type  are  available  on  all  of  the  CFA 
finalists.  The  evaluation  of  tools  in  this  category 
will  therefore  oe  restricted  to  sort/merge  programs 
and  editors  (interactive  source  language  editors, 
interactive  object  module  editors,  batch  source  language 
editors,  and  batch  object  module  editors). 

(6)  Information  Retrieval  Systems 

General  purpose  application  programs  operating  either 
on-line  (interactively)  or  in  the  batch  that  inter- 
pret user  requests  to  locate  and  display  information 
that  is  stored  either  within  a structured  database 
or  within  separate  files.  These  systems  can  be  classi- 
fied either  as  query  language  systems  or  as  report 
writers. 

(4)  Layer  1:  Operating  System  Services 

Layer  1 implements  the  operating  system  services  that  present  a "virtual 
machine"  interface  to  the  services/tools  at  layers  2 and  3 and  manage  the  real 
system  hardware.  The  primary  functions  of  the  layer  1 tools  are  listed  below: 
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( 1 ) Memo ry  Ma na  geme n t 
(2.)  Processor  Management 
( i)  Timer  Management 
(4)  Interrupt  Management 
(b)  Peripheral  Device  Management 
(b)  In[)ut/Uutput  Services 
(7)  PrograiM  hanoling  Services 
(b)  Task  ibdnagenent 
;4)  lns*’rui'ient  Jtion 
(iu)  Debugging  Services 
(li)  System  Startup  and  Restart 
lik)  .jOb  Control  facilities 
(li)  security 
(l4)  Exception  Handling 

The  layer  1 tool  types  are  generally  applicable  across  all  of  the  software 
development  activities,  -dyer  1 tool  types/capabilities  are  listed  oelow; 

U)  basic  operating  Systems 

Runs  single  user  processes  from  initiation  to  termina- 
tion. May  or  may  not  overlap  I/O  with  execution.  Pro- 
vides uasic  I/O  support  that  allows  user  to  refer  to 
tiles  syniool  i cal  ly  and  to  read  and  write  them  without 
knowing  the  hardware  details  of  the  I/O  Interface. 

Provides  basic  batch  supervisor  services  that  control 
nor.iial  and  abnormal  job  termination,  job  to  job  transi- 
tion, and  operator  communication.  Provides  a minimum 
base  r\.r  program  development  by  supporting  at  least 
one  language  translator  ano/or  linker/loader. 

(C)  -lul  ti programmi  ng  Operating  System 

provides  all  of  the  services  of  the  basic  Operating 
System.  Supports  the  concurrent  execution  of  two  or 
more  user  jobs  by  allowing  the  execution  of  any  job 
to  be  suspended  while  another  is  executed  without 
iny  special  programming  considerations  in  the  user 
jou.  Prevents  concurrently  executing  user  jobs  from 
accidentally  or  intentionally  destroying  each  other 
or  the  supervisor. 

(3)  I'iul  tiprocessor  Operating  System 

Allows  the  computing  load  to  be  spread  across  more 
than  one  processor  based  on  automatic  (programmed) 
load-leveling  algorithms  or  operator  control,  but 
does  not  require  special  case  programming  in  the 
user  job.  Multiprocessor  Operating  Systems  include 
the  shared  storage,  loosely  coupled,  and  networked 
types. 
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(4)  Virtual  Machine  Monitor 

The  operating  system  presents  an  interface  to  the 
user  program  that  makes  it  appear  that  the  program 
is  executing  on  a real  computing  system. 

(5)  Time-Sharing  Operating  Systems 

This  is  a variant  of  the  multi  programing  operating 
system  in  which  system  resources  are  allocated  to 
user  jobs  in  such  a way  that  all  jobs  appear  to 
progress  at  the  same  rate.  In  addition,  users  are 
allowed  to  "interact"  with  and  receive  output  from 
their  jobs  via  terminals.  Such  systems  are  optimized 
for  response  rather  than  throughput  or  equipment 
util ization. 

(6)  Real-Time  Operating  Systems 

Allows  user  jobs  to  be  executed  within  specified 
short  time  limits. 

(7)  Remote  Job  Entry 

A feature  of  operating  systems  that  allows  a batch 
job,  incluoing  its  input,  to  be  submitted  from  a 
device  that  is  connected  to  the  computer  system  by 
a teleprocessing  line.  The  device  may  be  a high 
speed  terminal  with  card  and  printer  devices,  a 
low  speed  terminal  with  a keyboard,  or  another 
computer. 

(B)  Checkpoint/Restart 

A feature  of  an  operating  system  that  allows  a 
user  program  to  periodically  have  its  state  saved 
so  that  it  may  be  restarted  from  a point  subsequent 
to  initiation  if  a hardware  or  software  failure 
occurs. 
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4.  EVALUATION  CRITERIA  OE  TriE  SOFTWARE  BASES 

a.  Uenoral 

One  of  the  funaa'iental  justifications  for  tne  entire  “-'CF  program  is  "soft- 
ware capture."  That  is  the  capture  of  the  support  software  investment  of  the 
winning  architecture.  In  orcer  to  coiiuare  the  three  finalist  archi tectures , a 
"roceOure  '.jas  r . •i''-'-  .Tori':  "Tir-^ify  t'^eir  suuoort  cottware  investment. 

"’■iO  fo:1c'..''nc  t..:-?!;  ’ te  , ■.  : •' 'ue  ■ f i at'' e . oofi' '■aol  - cc.lo 

00  f raws  I a :':a  int  ' j-  iters  are. 

a.  App1 i caoi I i ty 

0.  Availability  ty  '-rc("i tecture  manu^’acturer 

c.  Avail aoility  oy  other  than  Arcm tecture  Manufacturer 

Each  of  tnese  measures  is  aiscusseu  in  the  next  subsections.  Section  4.3 
■Jiscusses  the  consol i oati on  of  these  measures  into  financial  aata. 

b.  Inoivioual  measures 
(1)  App i 1 caoi 1 I ty 

Tnis  involves  aetenm  nation  of  the  functional  relevance  of  a given  software 
base  component  (tool  type)  to  tne  oevc-l opinent  of  military  tactical  software  sys- 
tems. Hppl  i caoi  1 1 ty  is  not  ifitt^nceo  ta  oe  a binary  criterion  but  shoulo  be  a mea- 
sure or  t'e  Tjte'  tia'  i .. p.^rt-rc-  v'  tne  component,  ranging  from  "not  applicable" 
to  "essential."  Factv.'r's  tn.t  sno.is  oe  considered  in  determining  tne  importance 
or  a tool  cy -■e,  in  a-jcit'?n  to  -.ssenti  al  i ty . induce  spectrum  coverage,  economic 

1 ipact.  software  377^^  dP.;  r.-.g  nuoper  of  different  instances  of  tne  same  tool  type 
fur  a given  ,.F-t.  esoers  AC-ro  allotted  o.uUO  points  and  asked  to  distribute  these 
joints  over  the  tools  t.'.'aryy^.  inci  eating  the  relative  appl  i caoi  1 i ty/importance  of 
each  tool.  If  a tool  is  rsse  tial  (in  a practical  ana  not  a tneoretical  sense)  for 
nilitary  co'.iputer  software  development  then  it  should  deserve  more  weight  than  one 
that  IS  only  nice  to  have.  Some  tools  are  dependent  on  others,  e.g.,  a librarian 
system  is  heavily  aepeniient  or  the  operating  system.  Therefore  the  assignment  of 

a high  weignt  to  the  librarian  systems  and  tne  assignment  of  no  weight  to  the  OS 
would  oe  inconsistent. 

Consiaeration  shoula  also  be  given  to  spectrum  coverage,  i.e.,  the  utility  of 
the  component  a(,ross  neve’opment  activities/phases  (requirements  analysis,  soft- 
ware design,  etc.)  as  well  as  across  development  disciplines  (management,  engineer- 
ing, aocumentation,  validation).  Consiaeration  should  be  given  to  economic  impact, 
ttie  potential  cost  savings  that  may  be  realized  through  use  of  the  tool  type,  or 
the  cost'  tnat  mignt  be  incurrea  shoulo  the  component  not  be  available.  Software 
size  iiiay  be  appliea  to  measure  the  latter.  Also,  a small  tool  type  (functionally) 
may  oesei^ve  loss  weight  than  a .iure  powerful  tool  type  ana  software  size  may  pro- 
vide some  measure  ot  the  uifferences. 

Table  3 depicts  the  ballot  which  was  sent  to  the  committee  members.  Upon 
receipt  of  the  ballots  the  results  will  be  ccmuileo  ana  applied  against  a preaeter- 
r.iined  tnresnola  value  ot  iuUU  points.  All  tools  which  fall  oelow  this  figure  in 
total  points  will  no  longer  oe  consiaerea.  The  justification  for  this  is  that  DoD 
could  not  afford  to  builo.  (in  fact  it  would  not  oe  wise  to  build),  tools  which  a 
''epresentati ve  spectrum  of  system  aevelopers  determined  were  not  applicable  in  a 
relative  sense. 
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TABLE  3 BALLOT  FOR  RELATIVE  APPLICABILITY 


1.  LAYER  3 

'1«1  Requirements  Analysis 

1.1.1  General  Purpose  Systen  Simulator 

1.1.2  System  Description  Language  & Analyzer 

1.2  Software  Design 


WEIGHTS 


1.2.1  Computer  Systen  Simulator 

1.2.2  Data  Base  Design  Aid 

1.2.3  Data  Dictionary  System 

1.3  Construct  Sv’stem  Tests 


1.3.1  Test. Data  Generator 
1.3*2  Test  Data  Auditor 
1.3*3  Test  Case  Design  Advisors 
1*3*3*1  F0RTR.AH 
1*3*3. 2 COBOL 
1*3*3*3  CM3-2  (1) 

1*3*3*^  C!>IS-2  (2) 

1*3*3*5  CM3-2  (3) 
l*3*3*o  JOVIAL  J3B 
1. 3*3*7  JOVIAL  J73 
1*3*3*8  tacpcl 
1*3. 3. 9 ATLAS 
1.3.3*30  OPAL 

1.3*^  Test  Instruments  & Analyzers 
(BY  LANGUAGE) 

1.4  Build  & Unit  Test 


1.4.1  Assemblers 

1.4. 1.1  Basic  Assembler 

1.4. 1.2  Macro  Assembler 

1.4.2  Compilers 
(BY  laj;guage) 

1.4.3  Linkers 

1.4. 3.1  Basic  Linker 

1.4. 3. 2 Simple  Overlay  Linker 

1.4. 3. 3 Extended  Overlay  Linker 


22 


TABLE  3 BALLOT  FOR  RELATIVE  APPLICABILITV 
(Continued) 


r 


1.U.4  Debugging  Aids 

1.4.U.1  Interactive  Sysbollc  Debugger 
(AS5E31ELER  & BY  HOL) 

1.4. 4. 2 Interactive  Absolute  Debugger 

1.4. 4. 3 "on-Interactive  S.vnbollc  Debugger 
(ASBUm-EI!  & BY  HOL) 

1.4. 4. 4 Ilon-Interactlve  Absolute  Debugger 
1.4.';  I.odulc  Libraries  & Change  Control  Systems 

1.4. 5.1  Basic  or  Loosely  Coupled 

1.4. 5. 2 Integrated  Library 

1»**^«5*3  Automatic  Software  Production 
& Test 

1.4.6  Perforannee  Monitors 

1.4. 6.1  Language  Dependent  Monitors 
(ASSE-'iBLER  & BY  KOL) 

1.4. 6. 2 Language  Independent  Monitor 

1.4.7  Standards  Enforcers 
(BY  LAilGUAGE) 

1.4.8  Preproces  sor  s /P.e  formatter 

1.4. 6.1  Preprocessors 
(by  LAIIGUAGE) 

1.4. 8. 2 Reformatters 
(by  LAIIGUAGE) 

2.  LAYER  2 

2.1  Data  Base  Management  System 

2.2  Pro.ject  Management  Aids 

2.2.1  PERT/CPM  System 

2.2.2  Project  Estimation  System 

2.3  Documentation  Aids 


2.3.1  Text  Processing  System 
2.3*2  Flowchart  Construction  Language 
(ASSE?IBLER  & BY  HOL) 

2.3*3  Automatic  Flowcharters 
(ASSEI'IBLER  & BY  iiOL) 
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TABLE'  3BALlOT  FOR  RELATIVE  APPLICABILITY 
(Continued) 

2.U  Data  Manipulation  Utilities 

2.4.1  Sort/Merge 

2.4.2  Editors 

2. 4. 2.1  Interactive  Source  Language 
Editors 

(ASSEIBLER  & BY  ROL) 

2. 4. 2. 2 Interactive  Object  Module 
Editors 

2.4.2. 3 Batch  Soiirce  Language  Editors 
(ASSEMBLER  & BY  HOL) 

2. 4. 2. 4 Batch  Object  Module  Editors 

2.5  Information  Retrieval  Systems 

2.5.1  Query  Language  System 

2.5.2  Report  Writer 

3.  LAYER  1 

3.1  BOS 

3.2  ms 

3.3  KPOS 

3.4  TSOS 

3.5  T503  + MPOS 

3.6  RTOS 

3.7  RTOS  + MOS 

3.8  RTOS  + MPOS 

3.9  RTOS  + TSOS 

3.10  BOS  + VMM 

3.11  JOS  +VMM 

3.12  MPOS  + VMM 

3.13  TSOS  + VI^-I 

3.14  T90S  + MPOS  + V'cM 
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(2)  Availability  from  Architecture  Manufacturers 

This  criterion  involves  Oetermi ni ng  »rfhether  the  support  software  tools  of 
Appendix  A are  availaole  from  the  finalist  architecture  manufacturers.  A tool 
will  oe  considered  available  if  it  is  presently  being  marketed  and  maintained. 
This  definition  permits  the  criterion  to  be  applied  uniformly  and  equitably  to 
all  the  finalist  manufacturers.  It  is  felt  that,  if  tools  under  development  were 
also  considered,  it  would  be  impossible  to  determine  whether  the  tool  was  one 
month,  one  year,  five  years  away,  etc.  from  oeing  marketed  by  the  company. 

The  actual  determination  of  availability  of  support  software  tools  will  be 
conducted  in  two  phases,  khase  I will  consist  of  providing  each  of  the  manufac- 
turers a list  of  the  support  software  tools  (Table  3)  as  well  as  the  descriptions 
of  Appendix  A and  requesting  them  to  answer  in  the  affirmative  for  all  tools 
which  they  actively  market  and  maintain.  Phase  II  will  consist  of  a visit  to 
each  of  the  manufacturers  by  government  personnel  and  an  independent  auditor  to 
obtain  supporting  documentation  and  to  audit  the  manufacturer  responses. 

At  tiie  end  of  Pnase  II,  an  accurate  list  of  available  support  software  for 
each  manufacturer  will  be  delineated. 

(3)  Availability  From  Other  Vendors 

It  was  felt  that  there  exists  a great  deal  of  support  software  available 
from  sources  other  than  the  architecture  manufacturers  which  meets  the  technical 
descriptions  of  Appenoix  A.  Such  software  should  be  included.  Again,  a firm 
criterion  is  needed  and  the  following  was  selected:  The  tool  must  meet  the  spe- 
cific requirements  of  tne  applicable  part  of  Appendix  A,  must  be  hosted  on  and 
targeted  for  one  of  the  finalist  architectures,  and  must  be  marketed  and  main- 
tained. The  International  Computer  Programs  Inc.  (ICP)  "INTERFACE  Reference 
Series"  has  been  chosen  as  the  source  document  because  it  has  become  a defacto 
standard  for  software  iiiarketing.  However  only  tools  which  were  not  marketed  by 
the  manufacturer  were  searched  for.  upon  finding  such  a tool  in  the  ICP  docu- 
ment, the  venoor  was  contacted  to  obtain  further  supporting  data. 

c.  Consolidation  of  Individual  Measures 

The  desired  end  product  of  the  entire  software  base  evaluation  is  a dollar 
figure  for  each  archi tecture' s existing  software  base,  as  well  as  a schedule 
depicting  the  cost  of  and  how  the  deficiencies  in  the  software  base  can  be  elim- 
inated, From  the  measures  discussed  in  Section  4.2,  lists  of  available  software 
tools  were  generated  for  each  architecture.  Also,  for  each  architecture,  a list 
of  software  deficiencies  were  generated  in  development  sequence.  The  develop- 
ment sequence  were  based  upon  applicability  and  PERT  ordering.  In  other  words 
if  tools  X,  Y,  and  1 are  rated  by  members  in  terms  of  applicability  in  the  order 
Z,  Y,  X,  but,  tiie  development  of  Y before  Z would  incur  a saving  in  overall  de- 
velopment costs,  then  the  final  development  sequence  would  be  ordered  as  Y,  Z,  X. 

Utilizing  these  figures,  an  overall  figure  for  the  software  base  of  each 
architecture  was  generated.  Also,  assunnng  a figure  of  2 million  dollars  a year 
as  an  estimate  of  the  support  software  RAU  dollars  available  when  the  MCF  pro- 
gram is  implemented,  and  utilizing  the  deficiency  lists  in  development  order, 
a development  schedule  was  generated  for  each  architecture  which  provides,  at 
a glance,  the  relative  future  deficiencies  of  each  architecture. 
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b.  RESULTS  OF  THE  EVALUATION  OF  THE  SOFTWARE  BASES 


a.  Results  by  Individual  Measures 

(1)  Applicability 

Early  in  May  1976,  the  Software  Evaluation  Methodology  Subcommittee  for- 
warded to  all  voting  members  of  the  Architecture  Selection  Committee  the  ballot 
depicted  in  Table  3 of  the  previous  section.  Each  member  was  requested  to  vote 
on  the  applicability  of  the  tools  delineated  in  that  Table  by  distributing  a max- 
imum of  5,000  points  over  the  tools. 

Table  4 contains  the  results  of  the  balloting.  There  were  a few  problems 
with  the  balloting,  the  most  notable  of  which  was  that  more  than  half  of  the  mem- 
bers failed  to  vote  on  particular  compilers,  but  cast  their  vote  for  compilers  in 
general.  Therefore,  it  was  decided  that  compilers  for  each  of  the  DoO  approved 
languages  as  follows  should  be  included  in  the  list  of  applicable  tools: 

FORTRAN 
COBOL 
CMS-2  (1) 

CMS-2  (2) 

CMS-2  (3) 

JOVIAL  J3B 
JOVIAL  J73 
TACPOL 

Automatic  test  equipment  languages  (ATLAS  and  OPAL)  were  removed  from  considera- 
tion since  none  were  available  for  the  finalist  architectures. 

As  can  be  seen,  the  threshold  of  1000  points  divides  the  list  approximately 
in  half  leaving  28  tools  to  be  utilized  in  the  availability  phase  of  the  evaluation. 

(2)  Availability  from  Architectures  Manufacturers 

In  order  to  determine  the  availability  of  support  software  tools,  two  steps 
were  taken.  First,  Table  3,  which  lists  all  of  the  support  software  tools  of  in- 
terest to  this  evaluation,  was  forwarded  to  the  three  finalist  architecture  manu- 
facturers along  with  the  Software  Evaluation  Methodology  Subcommittee's  Report 
entitled  "Procedure  for  Ranking  the  Software  Bases  of  Candidate  Architectures  for 
the  Military  Computer  Family."  Each  manufacturer  was  asked  to  read  the  report, 
especially  the  description  of  the  minimum  characteristics  of  the  tools  and  indicate 
which  of  the  Table  3 tools  are  marketed  and  maintained  by  the  company.  Table  5 
contains  the  results  of  that  process. 

Next  a meeting  was  set  up  by  the  chairman  of  the  Software  Evaluation  Method- 
ology Subcommittee  with  each  of  the  three  manufacturers.  Dr.  H.  Stone,  the  inde- 
pendent auditor,  ana  a representative  of  the  subcommittee  chairman  conducted  these 
meetings  and  requested  that  each  manufacturer  supply  supporting  documentation  to 
substantiate  their  claims  of  tool  existence/marketing.  Also,  each  manufacturer 
was  requested  to  provide  an  estimate  of  the  number  of  source  lines  of  code,  lan- 
guage utilized,  and  object  code  size  in  instructions  for  each  tool. 


r — ^ ^ 

I TABLE  4 

f COMPILED  RESULTS  OF  THE  APPLICABILITY  BALLOTING 


9708 

4405 

4317 

3752 

3367 

3010 

2969 

2687 

2677 

2380 

2270 

1872 

1688 

1645 

1623 

1535 

1418 

1400- 

1390 

1310 

1217 

1187 

1158 

1140 

1110 

1095 

1003 

1006 


Compilers 
Macro  Assemblers 

Interactive  Source  Language  Editors 
Interactive  Symbolic  Debuggers 
Extended  Overlay  Linker 
Test  Case  Design  /advisors 

Integrated  Library  : 

Text  Processing  System  1 

DBMS  j 

GP  System  Simulator  j 

TSOS  + VMM 

Language  Independent  Monitors 

Test  Data  Generator  i 

Non-Interacti ve  Symbolic  Debugger  ] 

Computer  System  Simulator  I 

Batch  Source  Language  Editors  | 

Language  Dependent  Monitors  i 

TSOS  + MPOS  + mA  j 

Basic  Assembler  i 

RTOS  + TSOS 

Test  Instrumenters  & Analyzers 
Automatic  SL'  Production  & Test 
Basic  Linker 
Standards  Enforcers 
Reforniatters 
Test  Data  Auditor 

Simple  Overlay  Linker  i 

Data  Base  Design  Aid 


975 

880 

855 

834 

808 

795 

792 

760 

757 

705 

675 

658 


Sort/Merge 

Preprocessors 

Basic  or  Loosely  Coupled  Library 
Project  Estimation  System 
System  Description  Language  & Analyzer 
RTOS 

PERT/CPM 

TSOS 

MOS 

BOS 

Report  Writers 

Interactive  Object  Module  Editors 
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TABLE  4 


COMPILED  RESULTS  OF  THE  APPLICABILITY  BALLOTING 
(Continued) 


642 

Interactive  Absolute  Debugger 

622 

Data  Dictionary  System 

615 

RTOS  + HPOS 

607 

Query  Language  Systems 

600 

PDL 

600 

Batch  Object  Module  Editors 

600 

Automatic  Flowcharters 

558 

Flowchart  Construction  Languages 

513 

Non-Interactive  Absolute  Debugger 

410 

MPOS 

365 

TSOS  + MPOS 

350 

BOS  + VMM 

315 

MOS  + VMM 

305 

RTOS  + MOS 

250 

MPOS  + VMM 

o 

m 

o 


1,1.1  General  Purpose  System  Simulator 
1.2  Software  Design 


1.2.1  Computer  System  Simulator 

1.2.2  Data  Base  Design  Aid 

1.3  Construct  System  Tests 


1.3.1 

1.3.2 

1.3.3 


Test  Data  Generator 
Test  Data  Auditor 
Test  Case  Design  Advisors 


1.3. 3.1 

1.3. 3. 2 


FORTRAN 

COBOL 


1.3. 3. 3 CMS-2  (1) 

1.3. 3. 4 CMS-2  (2) 

1.3. 3. 5 CMS-2  (3) 

1.3. 3. 6 JOVIAL  J3B 

1.3. 3. 7 JOVIAL  J73 

1.3. 3. 8 TACPOL 

1.3.4  Test  Instruments  & Analyzers 
(BY  LANGUAGE) 

1 .4  Build  & Unit  Test 


N N 
N N N 
N N N 
N N N 
N N N 
N N N 
N N N 


N 

'N  I N I N 


1.4.1  Assemblers 

1.4. 1.1 

Basic  Assembler 

1 .4.1 .2 

Macro  Assembler 

1.4.2  Compilers 

1.^.2.1 

FORTRAN 

1.4. 2. 2 

COBOL 

1.4. 2. 3 

CMS-2  (1) 

1.4. 2. 4 

CMS-2  (2) 

1.4. 2. 5 

CMS-2  (3) 

1.4. 2. 6 

JOVIAL  J3B 

1.4. 2. 7 

JOVIAL  J73 

1.4. 2. 8 

TACPOL 

N N N 


TABLE  5 SOFTWARE  AVAILABILITY  AS  CITED  BY  THE  MANUFACTURERS 

(Continued) 

1 .4.3  Linkers  . . 


1.4. 3.1  Basic  Linker 

Y 

Y 

Y 

1.4. 3. 2 Simple  Overlay  Linker 

N 

Y 

Y 

1.4. 3. 3 Extended  Overlay  Linker 

r 

Y 

1.4.4  Debugging  Aids 

1.4. 4.1  Interactive  Symbolic  Debugger 

(ASSEMBLER  & BY  HOL) 

Y 

N 

N 

1.4. 4. 3 Non-Interactive  Symbolic  Debugger 

Y ! 

N 

N - 

(ASSEMBLER  & BY  HOL) 

1 

1.4.5  Module  Libraries  & Change  Control  Systems 

1.4. 5. 2 Integrated  Library 

N 

N 

Y 

1.4. 5. 3 Automatic  Software  Production 

& Test 

N 

N 

Y 

1.4.6  Performance  Monitors 

1.4. 6.1  Language  Dependent  Monitors 

(ASSEMBLER  & BY  HOL) 

N 

..N.. 

N 

1.4. 6. 2 Language  Dependent  Monitor 

N 

N 

IN 

1.4.7  Standards  Enforcers 

(BY  LANGUAGE) 

1.4.8  Preprocessors/Reformatter 

N 

N 

N 

1.4. 8. 2 Reformatters 

(BY  LANGUAGE) 

N 

N 

N 

LAYER  2 

2.1  Data  Base  Management  System 

Y 

[ Y 

N 

2.3  Documentation  Aids 

1 

■ 

2.3.1  Text  Processing  System 

V 

Y 

H 

2.4  Data  Manipulation  Utilities 

2.4.2  Editors 

2. 4. 2.1  Interactive  Source  Language 

Editors  (ASSEMBLER  & BY  HOL)  Y 

Y 

Y 

2. 4. 2. 3 Batch  Source  Language  Editors 

1 

(ASSEMBLER  & BY  HOL) 

Y 

i Y 

Y 

LAYER  1 

i 

3.9  RTOS  + TSOS 

V 

' Y 

N 

3.13  TSOS  + V^M 

Y 

N 

N 

3.14  TSOS  + MPOS  + VMM  30 

Y' 

‘N 

Dr.  H.  Stone  has  documented  this  audit  which  is  included  as  Appendix  B to  this 
report.  Table  6 contains  the  result  of  the  audit. 

(3)  Availability  from  Other  Vendors 

Software  tools  which  meet  the  technical  requirements  of  Section  3.4  and 
were  marketed  by  other  than  the  manufacturers  also  were  investigated.  The 
International  Cotiiputer  Programming  Inc.  (ICP)  "INTERFACE  Reference  Series," 

Jan  197b,  was  utilized  as  the  source  document  for  this  investigation.  Tables 
7-9  list  the  support  software  tools  by  tool  type.  Also  listed  is  the  mar- 
keting company  as  well  as  the  one  installation  sale  or  yearly  lease  cost. 

Each  of  the  companies  was  contacted  and  requested  to  provide  further  detailed 
documentation  on  the  product.  As  stated  earlier,  if  an  architecture  manufac- 
turer marketed  a particular  tool,  that  tool  was  not  sought  in  the  ICP  document. 
Table  10  depicts  an  overall  composite  of  software  availability  from  sources 
other  than  the  architecture  manufacturer.  The  dashes  indicate  tools  which  the 
manufacturer  uia  iiiarket  and  therefore  were  not  looked  for  in  the  ICP  document. 


b.  Consol i oati on  of  Results 

The  main  purpose  of  the  Software  Evaluation  was  to  determine  the  relative 
current  software  collar  investment  of  the  three  architectures.  Also  it  was  de- 
sired to  Obtain  the  relative  deficiencies  of  the  architectures  in  terms  of  dol- 
lars. In  order  to  do  this,  a development  cost  for  each  tool  was  needed.  It  was 
known  that  the  software  vendors  considered  such  information  to  be  proprietary. 
Therefore,  the  following  approach  was  taken:  First  each  manufacturer  was  re- 
quested to  provioe  the  source  code  size  for  his  available  tools  as  well  as  the 
language  which  the  tool  was  written  in,  ana  the  object  code  size  in  instructions. 
Second,  a productivity  figure  was  needed.  F.  Brooks',  "The  Mythical  Man-Month" 
was  considered  to  be  the  best  source  since  it  compiled  productivity  figures  from 
the  IBM's  0S/3bu  development  as  well  as  Bell  Labs'  ESS  software  development. 
Brooks  cites  buu  lines  of  cooe  per  man-year  for  operating  system  development  and 
I!,uuu  lines  of  code  per  man-year  for  other  support  software  development.  It  is 
felt  that  the  state-of-the-art  in  operating  system  has  improved  significantly 
since  his  data  was  obtained  (nearly  ten  years  ago)  and  thus  a figure  of  l,00u 
and  I^,uuu  lines  of  code  per  man-year  for  operating  system  and  other  support  soft- 
ware, respectively,  was  decided  upon. 

It  should  be  noted  that  anticipated  maintenance  costs  of  support  software 
have  not  been  estimated  in  this  analysis.  A question  could  be  raised  as  to 
accuracy  of  the  analysis  without  this  consideration,  however,  it  was  felt  that 
it  would  be  impossible  to  estimate  accurate  costs  for  the  maintenance  of  the 
support  software  tools  because  many  of  these  tools  developed  under  the  MCF  pro- 
gram may  be  adopted,  niarketed  ana  maintained  by  the  selected  architecture  manu- 
facturer or  by  the  developer  of  the  tools. 

Table  11  aepicts  the  source  lines  of  code  as  supplied  by  the  manufacturer. 

As  can  be  seen,  this  data  is  scarce  and  therefore  estimates  had  to  be  made  of 
the  cost  of  tools  for  which  there  was  no  data.  For  all  tools,  a figure  of 
$7U,000  a man-year  was  utilized. 
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TABLE  6 SOFTWARE  AVAILABILITY  AS  DETERMINED  BY 
AUDIT  OF  MANUFACTURERS  RESPONSES 
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TABLE  6 SOFTWARE  AVAILABILITY  AS  DETERMINED  BY  AUDIT  OF 
MANUFACTURERS  RESPONSES  (Continued) 


1.4.4 


1.4.5 


1.4.6 


1.4.7 


1.4.8 

I 


i 

1 


Debugging  Aids 

1.4. 4.1  Interactive  Symbolic  Debugger 


1.4. 4. 1.1  Assembler 

’ N ! 

N 

N ' 

1.4. 4. 1.2  FORTRAN 

1 Y ■ 

N 

N 

1.4. 4. 1.3  COBOL 

1 * 

N 

N i 

1.4.4. 1.4  CMS-2 

? N 

N 

N 

1.4.4. 1.5  JOVIAL 

: N 

N 

N 1 

1.4.4. 1.6  TACPOL 

, N 

N ■■ 

N 1 

1.4. 4. 3 

Non-Interacti ve  Symbolic  Debuggers 

I 

! 

1.4. 4. 3.1  Assembler 

. N 

N 

N 

1.4. 4. 3. 2 FORTRAN 

Y 

Y 

'n 

1.4. 4. 3. 3 COBOL 

Y 

Y 

N 

1.4.4. 3.4  CMS-2 

. N 

N 

N ' 

1.4. 4. 3. 5 JOVIAL 

. N 

N 

N 

1.4. 4. 3. 6 TACPOL 

, H 

'T” 

Module  Libraries  & Change  Control  Systems 

; 

1.4;5.2 

Integrated  Library 

' N 

N 

N I 

1.4. 5. 3 

Automatic  Software  Production  & Test 

N 

_N 

N 

Performance  Monitors 

1.4. 6.1 

Language  Dependent  Monitor 

; 

i i 

1 

1.4. 6. 1.1  Assembler 

■ N 

N 

N 

1.4.6. 1.2  FORTRAN 

Y 

N 

-fT-  ^ 

1.4.6. 1.3  COBOL 

Y “ 

“■'N'  ■ 

N'~ 

1.4.6. 1.4  CMS-2 

N 

N 

N* 

1.4.6. 1.5  JOVIAL 

N 

^ N 

N' 

1.4.6. 1.6  TACPOL 

N 

, N 

N 

1.4. 6. 2 

Language  Independent  Monitor 

N ; N 

N 

Standards  Enforcers 


1.4.7. 1 

F0RTPJ\N 

N 1 

N 

N ' 

1.4. 7. 2 

COBOL 

N 

N 

N 

1.4. 7. 3 

CMS-2 

N_j 

,.N. 

N 

1.4. 7.4 

JOVIAL 

j 

N 

N 

1.4. 7.5 

TACPOL 

N ^ 

N 

N 

Preprocessors,  Reformatters 

r'  ’ 

1 

i 

{ 

1.4. 8. 2 

Reformatters 

1 

1.4. 8. 2.1 

FORTRAN 

N ’ 

' N 

N 

1.4. 8. 2. 2 

COBOL 

N 

i N 

N 

1.4. 8. 2. 3 

CMS-2 

N 

1 N 

N 

1.4.8. 2.4 

JOVIAL 

~Tr 

"N" 

1.4. 8. 2. 5 

TACPOL 

N 

i N 

N 

i 

1 

I 


I 


TABLE  6 SOFTWARE  AVAILABILITY  AS  DETERMINED  BY  AUDIT 
OF  MANUFACTURERS  RESPONSES  (Continued) 


1 

i TABLE  7 - SOFTWARE  MARKETED 

^ * BY 

' OTHER  THAN  IBM 

I FOR  THE 

i IBM  360/370  ARCHITECTURE 


1.2.1  Computer  System  Simulator 

a.  SCERT-76 

Marketed  by  COMTEN,  Inc.,  Rockville,  MD  $22,000 

b.  CASE 

Marketed  by  TESTDATA  Systems  Corporation,  McLean,  VA  $17,000 

1.3.1  Test  Data  Generator 

a.  DATAMiACS 

Marketed  by  Management  & Computer  Services,  Inc. 

Valley  Forge,  PA  $ 8,500 

b.  TDG-L  Test  Data  Generating  Language 

Marketed  by  K&A  Software  Products,  Dal  lax,  TX  $11,000 

1.3.2  Test  Data  Auditor 

a.  APRO/Test  File  Checker 

Marketed  by  Synergistics  Corporation,  Burlington,  PA  $ 3,800 
1.4.4. 1.1  Interactive  Symbolic  Debugger  (Assembler) 
a.  SYMBUG-A 

Marketed  by  Standard  Data  Corporation,  NY,  NY  $17,500 

1.4. 5. 2 Integrated  Library 

a.  PULMACS  III 

Marketed  by  Management  & Computer  Services,  Inc., 

Valley  Forge,  PA  ^ $ 3,000 

b.  SUR  (Software  Utility  Routine)  ^ 

Marketed  by  Boeing  Computer  Ser\/ices,  Inc., 

Seattle,  WA  $ 5,000 

c.  The  LIBRARIAN 

Marketed  by  Applied  Data  Research,  Inc. , Princeton,  NJ  $ 5,300 

d.  PANVALET 

Marketed  by  Pansophic  Systems,  Inc.  Oakbrook,  IL 
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■ TABLE  1 - SOFTWARE  MARKETED 

BY 

: OTHER  THAN  IBM 

IBM  360/370  ARCHITECTURE 
(Continued) 

I 1.4. 6. 1.1  Language  Dependent  Monitor  (Assembly) 

' a.  STROBE 

I Marketed  by  Programart  Corporation,  Cambridge,  MA  $ 9,400 

I 1.4. 6. 2 Language  Independent  Monitor 

a.  Problem  Program  Evaluator  - PPE 

Marketed  by  Broole  & Baggage,  Sunnyvale,  CA  $ 9,800 

! 

? b.  Operating  System  Monitor 

! Marketed  by  Lincoln  Land  Software  Systems,  Inc., 

f Springfield,  IL  $2, 400/first  install- 

ation 

1 ,200/each  additional 
installation 

1.4. 8. 2.1  Reformatter  (FORTRAN) 

a.  IFTRAN-2  FORTRAN  Preprocessor  for  Structured 
Programming 

Marketed  by  General  Research  Corporation, 

Santa  Barbara,  CA  $ 1,000 

1.4. 8. 2. 2 Reformatter  (COBOL) 

a.  Reformat-Automatic  Standardization  of  COBOL 
Source  Programs,  Plus  Shorthand  COBOL 

Marketed  by  EDP  Management,  Inc.,  LaMesa,  CA  $ 3,500 


TABLE  8 - SOFTWARE  MARKETED 
BY 

OTHER  THAN  DEC 
FOR  THE 

PDP-11  ARCHITECTURE 


1.3. 4.1  Test  Instrumenter  & Analyzer  (FORTRAN) 

a.  RXVP-1  Automated  Verification  System  for  FORTRAN 
Marketed  by  General  Research  Corporation 

Santa  Barbara,  CA 

b.  TAP  Testing  Analysis  Package 

Marketed  by  General  Research  Corporation 
Santa  Barbara,  CA 

1.4. 4. 1.3  Interactive  Symbolic  Debugger  (COBOL) 

a.  Dependable  Debugging  Tool 

Marketed  by  Innovative  Systems  Association 
New  York,  NY 

1.4. 8. 2.1  Reformatter  (FORTRAN) 

a.  IFTRAN-2  FORTRAN  Preprocessor  for  Structured 
Programming 

Marketed  by  General  Research  Corporation 
Santa  Barbara,  CA 


$14,000 


$ 7,500  lease 
only 


$225.00 


$1 ,000 
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TABLE  9 - SOFTWARE  MARKETED 
BY 

OTHER  THAN  INTERDATA 
FOR  THE 

INTERDATA  8/32  ARCHITECTURE 


1.3. 4.1  Test  Instrumenter  & Analyzer  (FORTRAN) 

a.  RXVP-1  Automated  Verification  System  for  FORTRAN 
Marketed  by  General  Research  Corporation 

Santa  Barbara,  CA 

b.  TAP  - Testing  Analysis  Package 
Marketed  by  General  Research  Corporation 

Santa  Barbara,  CA 

1.4. 8. 2.1  Reformatter  (FORTRAN) 

a.  IFTRAN-2  FORTRAN  Preprocessor  for  Structured 
Programming 

Marketed  by  General  Research  Corporation 
Santa  Barbara,  CA 


$14,000 

$ 7, 500/ lease  oniy 


$ 1,000 


( 
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TABLE  10  COMPOSITE  OF  SOFTWARE 
AVAILABILITY  FOR  THE  IBM,  DEC  AND 
INTERDATA  ARCHITECTURES  FROM  OTHER  SOURCES 
THAN  IBM,  DEC  AND  INTERDATA 
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TABLE  10  COMPOSITE  OF  SOFTWARE 
AVAILABILITY  FOR  THE  IBM,  DEC  AND 
INTERDATA  ARCHITECTURES  FROM  OTHER  SOURCES 
THAN  IBM,  DEC  AND  INTERDATA 
(CONTINUED) 
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TABLE  10  COMPOSITE  OF  SOFTWARE 
AVAILABILITY  FOR  THE  IBM,  DEC  AND 
IMTERDATA  ARCHITECTURES  FROM  OTHER  SOURCES_ 
THAN  IBM,  DEC  AND  INTERDATA 

(CONTINUED)  i 


1,4. 8, 2,1  FORTRAN 
1,4, 8, 2. 2.  COBOL 

1.4. 8. 2. 3 CMS-2 

1.4. 8. 2. 4 JOVIAL 

1.4. 8. 2. 5 TACPOL 


2.  LAYER  2 


' s! 

■ 3 , 

Y I 

Y 
N 

N !. 

N I 

t 

I 

i 


o 

m 

o 


f-i 


Y_,Y 
N_  U 
H "'N 


2.1 

Data  Base  Management  System 

j — j.  - 

; - * - 

« 

, N 

2.3 

Documentatipn  Aids 

i ; 

! i 

i 

i 

1 

2,3,1  Text  Processing  System 

I ■ \ ■ 

1 t 

i 

S N 

2.4 

Data  Manipulation  Utilities 

i i 

1 . 

j 

I 

2,4,2  Editors 

; 

* 

; 

2.4,2. 1 Interactive  Source  Language 

• . . 

' 

Editors 

2, 4. 2. 3 Batch  Source  Language 

- 

Editors 

• - c 

LAYER  1 

i 

i 

3.9 

RTOS  + TSOS 

• « 

• 

3.13 

TSPS  + VMM 

, - N 

N 

3.14 

TSOS  + MPOS  + VMM 

- N 

_N_ 

41 


A 


INTERDATA 


r 


TABLE  n - ESTIKATLS  Cf  SIZE 

BY 

SOURCE  LIMES  OF  CO  . 

AUD 

ESTIMATED  TOOL  COST 
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TABLE  11  - ESTIMATES  OF  TOOLS  SIZE 
BY 

SOURCE  LINES  OF  CODE 
AND 

ESTIMATED  TOOL  COST 
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j210K  '2IOK 

1.4.7  Standard  Enforcers 

1 

420K  420K 

i 

(for  all  (for  all 

1.4. 7.1  FORTRAN 

Ian-  languages) 

1.4. 7. 2 COBOL 

guages)  , 

1.4. 7. 3 CMS-2 

1.4. 7. 4 JOVIAL 

1 

1.4. 7. 5 TACPOL 

1 

1.4.8  Reformatters 

1 

560K  560K 

(for  all  (for  all 

1.4.B.1  FORTRAN 

Ian-  languages) 

1.4. 8. 2 COBOL 

iguages)  , 

1.4. 8. 3 CMS-2 

1 

1.4. 8. 4 JOVIAL 

1 1 

1.4. 8. 5 TACPOL 

i 

2.1  Data  Base  Management  System 

119K 

i 42,000K 

Assembly 

i 

Language 

l32K 

1 1 

1 

iRun-time  i 

1 

'Words  ' 

! 1 

1 I 
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TABLE  n - ESTIV'.TES  OF  TOOLS  SIZE  ] 

BV 

SOURCE  LINES  OF  CODE 
AND 

ESTIMATED  TOOL  COST 
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Tables  12  - 14  depict  the  software  bases  for  the  IBM  360/370, 

DEC  PDP-11  and  the  Interdata  8/32.  Tables  15  - 17  list  the  deficiencies 
of  each  architecture  and  proposed  development  sequence  as  well  as  the 
estimated  cost  of  developing  each  tool. 

Tables  18  - 20  depict  schedules  for  software  base  deficiency  correc- 
tion of  the  three  architectures  based  upon  an  estimated  expenditure  of 
$2,000,000  a year,  while  Tables  21  - 23  depict  similar  schedules  based  upon 
$3,000,000  a year  and  Tables  24  - 26  depict  schedules  based  upon 
$1 ,000,000  a year. 


TABLE  1?-  IBM  360/370  SOFTWARE  BASE 


1. 

2. 

3. 

4. 

5. 

6. 

7. 

8. 

9. 

10. 
'll. 

12. 

13. 

14. 

15. 

16. 

17. 

18. 


General  Purpose  System  Simulator 
Computer  System  Simulator 
Data  Base  Design  Aid 
Test  Data  Generator 
Test  Data  Auditor 

Test  Insturmenter  & Analyzer  (FORTRAN,  COBOL) 

Basic  Assembler 
Macro  Assembler 

Compiler  (FORTRAN,  COBOL,  JOVIAL) 

Basic  Linker 

Simple  Overlay  Linker 

Extended  Overlay  Linker 

Interactive  Symbolic  Debugger  (ASSEMBLER,  FORTRAN, 

COBOL) 

Non-Interactive  Symbolic  debugger  (FORTRAN,  COBOL) 
Integrated  Library 

Language  Dependent  Monitor  (ASSEMBLER,  FORTRAN, 

COBOL) 

Language  Independent  Monitor 
Reformatter  (FORTRAN,  COBOL) 


Estiitrcted  Cost 
700K  ^ ' 
420K 
1150K 
350K 
140K 
BOOK 
280K 
1400K 
1 1 ,600 

210K 
21  OK 
b60K 

700K 

90K 

700K 

525K 

210K 

224K 
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TABLE  12  - IBM  360/370  SOFTWARE  BASE 
(Continued) 


19. 

Data  Base  Management  System 

Estimated  Cost 

4200K 

20. 

Text  Processing  System 

630K 

21. 

Interactive  Source  Language  Editors 

560K 

22. 

Batch  Source  Language  Editors 

210K 

23. 

RTOS  + TSOS 

2500K 

24. 

TSOS  + VMM 

2800K 

25. 

TSOS  + MPOS  + VMM 

1400K 

TOTAL  32,269K 


r 

TABLE  13-  DEC  PDP-11  SOFTWARE  BASE 


Estimated  Cost 

1. 

Data  Base  Design  Aid 

1150K 

2. 

Test  Insturmenter  & Analyzer  (FORTRAN) 

280K 

3. 

Basic  Assembler 

280K 

4. 

Macro  Assembler 

1400K 

5: 

Compiler  (FORTRAN,  COBOL) 

9600K 

6.'" 

Basic  Linker 

210K 

7. 

5IMRC  Overlay  Linker 

210K 

8. 

Extended  Overlay  Linker 

560K 

9. 

Interactive  Symbolic  Debugger  (COBOL) 

230K 

10. 

Non-Intei  active  Symbolic  Debugger  (FORTRAN, 

COBOL)  90K 

11. 

Reformatter  (FORTRAN) 

IlOK 

12. 

Data  Rase  Management  System 

420CK 

13. 

Text  Processing  System 

630K 

14. 

Interactive  Source  Language  Editor 

560K 

15. 

Batch  Source  Language  Editor 

210K 

16. 

RTOS  + TSOS 

?Rnnif 

TOTAL 

22.220K' 
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TABLE  14-  INTERDATA  8/32  SOFTWARE  BASE 


I HWWJBPiPlH 


Estimated  Cost 


1. 

Test  Instrumenters  & Analyzers  (FORTRAN) 

280K 

2. 

Basic  Assembler 

280K 

3. 

Macro  Assembler 

1400K 

4. 

Compilers  (FORTRAN,  COBOL) 

9600K 

5. 

Basic  Linker 

210K 

6. 

Simple  Overlay  Linker 

210K 

7. 

Reformatter  (FORTRAN) 

IlOK 

8. 

Interactive  Source  Language  Editor 

560K 

9. 

Batch  Source  Language  Editor 

210K 

10. 

RTOS  + TSCS 

2500K 

TOTAL 

15,360K 

j TABLE  15  - COMPOSITE  SOFTWARE  DEFICIENCIES 

OF 

IBM  360/370  ARCHITECTURE 

; (ORDERED  IN  PROPOSED  DEVELOPMENT  SEQUENCE) 


Estimated  Cost 


1.  Compilers  - CMS-2,  TACPOL  3500K 

2.  Interactive  Symbolic  Debugger  - CMS-2,  JOVIAL,  TACPOL  700K 

3.  Non-Interactive  Symbolic  Debugger  - ASSEMBLER  180K 

CMS-2 

JOVIAL 

TACPOL 

4.  Test  Case  Design  Advisor  - FORTRAN  2100K 


COBOL 

CMS-2 

JOVIAL 

TACPOL 

5.  Language  Dependent  Monitors  - CMS-2  525K 

JOVIAL 

TACPOL 

6.  Standards  Enforcers  - FORTRAN  420K 

COBOL 

CMS-2 

JOVIAL 

TACPOL 

7.  Reformatters  - CMS-2  330K 

JOVIAL 

TACPOL 

8.  Test  Instrumenters  & Analyzers  - CMS-2  840K 

JOVIAL 

TACPOL 

9.  Automatic  Software  Production  & Test  lOOOK 


TOTAL  9,595K 
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TABLE  16-  COMPOSITE  SOFTWARE  DEFICIENCIES 

OF 

DEC  PDP-11  ARCHITECTURE 
(ORDERED  IN  PROPOSED  DEVELOPMENT  SEQUENCE) 


Estimated  Cost 


1.  Compilers  - CMS-2  4900K 

JOVIAL 

TACPOL 

2.  Integrated  Library  700K 

3.  Interactive  Symbolic  Debugger  - ASSEMBLER  1200K 

FORTRAN 

CMS-2 

JOVIAL 

TACPOL 

4.  Non-Interactive  Symbolic  Debugger  - ASSEMBLER  180K 

CMS-2 

JOVIAL 

TACPOL 


5.  Test  Case  Design  Advisor  - FORTRAN  2100K 

COBOL 

CMS-2 

JOVIAL 

TACPOL 

6.  Test  Data  Generator  350K 

■/.  Language  Independent  Monitor  210K 

8.  Language  Dependent  Monitor  - ASSEMBLER  1050K 


FORTRAN 

COBOL 

CMS-2 

JOVIAL 

TACPOL 


9.  General  Purpose  Sy'^tem  Simulator  700K 

10.  Computer  System  Simulator  420K 


TABLE  16  - COMPOSITE  SOFTWARE  DEFICIENCIES 

OF 

DEC  PDP-n  ARCHITECTURE 
(ORDERED  IN  PROPOSED  DEVELOPMENT  SEQUENCE) 
(Continued) 


Estimated  Cost 


11.  Standards  Enforcers  - FORTRAN 

COBOL 

CMS-2 

JOVIAL 

TACPOL 

12.  Test  Data  Auditor 

13.  Reformatters  - COBOL 

CMS-2 

JOVIAL 

TACPOL 

14.  Test  Instrumenters  & Analyzers  - COBOL 

CMS-2 

JOVIAL 

TACPOL 

15.  Automatic  Software  Production  & Test 

16.  TSOS  + VMM 

17.  TSOS  MPOS  + VMM 


TOTAL 


1120K 


lOOOK 

2800K 

lAOOK 


19,130K 


TABLE  17-  COMPOSITE  SOFTWARE  DEFICIENCIES 

OF 

INTERDATA  8/32  ARCHITECTURE 
(ORDERED  IN  PROPOSED  DEVELOPMENT  SEQUENCE)  . 


Estimated  Cost 

1.  Compilers  - CMS-2  4900K 

JOVIAL 

TACPOL 

2.  Integrated  Library  700K 

3.  Entended  Overlay  Linker  560K 

4.  Interactive  Symbolic  Debugger  - ASSEMBLER  1400K 

FORTRAN 

COBOL 

CMS-2 

JOVIAL 

TACPOL 

5.  Non-Interactive  Symbolic  Debugger  - ASSEMBLER  280K 

FORTRAN 

COBOL 

CMS-2 

JOVIAL 

TACPOL 

6.  Test  Case  Design  Advisor  - FORTRAN  2100K 

COBOL 

CMS-2 

JOVIAL 

TACPOL 


7. 

Text  Processing  System 

630K 

8. 

Test  Data  Generator 

350K 

9. 

Language  Independent  Monitor 

210K 

10.  Language  Dependent  Monitor  - ASSEMBLER 

FORTRAN 

COBOL 

CMS-2 

JOVIAL 

TACPOL 

54 


1050K 

i 


J 


TABLE  17  - COMPOSITE  SOFTWARE  DEFICIENCIES 

OF 

INTERDATA  8/32  ARCHITECTURE 
(ORDERED  IN  PROPOSED  DEVELOPMENT  SEQUENCE) 
(Continued) 


11.  Data  Base  Management  System 

12.  Data  Base  Design  Aid 

13.  General  Purpose  System  Simulator 
Computer  System  Simulator 


14 


15.  Standards  Enforcers  - FORTRAN 

COBOL 

CMS-2 

JOVIAL 

TACPOL 


16.  Test  Data  Auditor 


17.  Reformatters  - COBOL 

CMS-2 

JOVIAL 

TACPOL 


18.  Test  Instrumenter  & Analyzer  - COBOL 

CMS-2 

JOVIAL 

TACPOL 


19.  Automatic  Software  Production  & Test 

20.  TSOS  + VMM 

21 . TSOS  + MPOS  + VMM 


TOTAL 


Estimated  Cost 


4200K 

1150K 

700K 

420K 

420K 


140K 

440K 


1120K 


lOOOK 

2800K 

1400K 


25,970K 


TABLE  18  - SCHEDULE  FOR  CORRECTION  OF  IBM  360/370 

SOFTWARE  DEFICIENCIES  (based  upon  $2  million  per  year) 
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TABLE  20  - SCHEDULE  FOR  CORRECTION  OF  INTERDATA  8/32 


n.  Data  Base  Management  System 


TABLE  21  - SCHEDULE  FOR  CORRECTION  OF  IBM  360/370  SOFTWARE  DEFICIENCIES 
(BASED  UPON  $3  million  per  year) 


TABLE  22  - SCHEDULE  FOR  CORRECTION  OF  DEC  PDP-ll  SOFTWARE  DEFICIENCIES 
(Based  upon  S3  million  per  year) 


11.  Standard  Enforcers 
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TABLE  23  - SCHEDULE  FOR  CORRECTION  OF  INTERDATA  8/32  SOFTWARE  DEFICIENCIES 


TABLE  24  - SCHEDULE  FOR  CORRECTION  OF  IBM  360/370  SOFTWARE  DEFICIENCIES 


10.  Computer  System  Simulator 
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HEDULE  FOR  CORRECTION  OF  INTERDATA  8/32  SOFTWARE 


TABLE  26  - SCHEDULE  FOR  CORRECTION  OF  INTERDATA  8/32  SOFTWARE  DEFICIENCIES 
(based  upon  $1  million  per  year)  (Continued) 


APPENDIX  A - DESCRIPTION  OF  SOFTWARE  BASE  COMPONENTS 


A,1  Introduction 

This  appendix  contains  descriptions  of  each  of  the  software  tool  categories 
recommended  for  inclusion  in  the  evaluation  of  the  software  base.  One  purpose 
of  these  descriptions  was  to  provide  the  Selection  Committee  with  a sufficient 
definition  to  decide  on  the  merits  of  the  tool  when  allocating  applicability 
points  and  evaluating  the  availability  of  tools.  The  descriptions  attempt  to 
establish  a minimum  level  of  features  or  capabilities  required  of  a tool  in  order 
for  it  to  qualify  for  acceptance.  The  intent  is  to  establish  a threshold  to 
differentiate  usable  tools  from  those  that  are  not. 

The  tool  descriptions  are  organized  as  illustrated  in  Tables  A-1  to  A-3 
along  the  structure  presented  in  Section  3.3.  First,  the  layer  3 tools  are  pre- 
sented grouped  under  primary  phase  of  the  development  process  which  the  tools 
support.  Next,  the  layer  2 tools  are  presented  grouped  under  the  functions  that 
they  support.  Layer  2 functions  are  generally  common  to  all  the  development 
activities/phases.  Finally,  selected  features  of  layer  1 tools  (operating  sys- 
tems) are  discussed.  The  operating  systems  features  selected  represent  a judg- 
ment of  high  cost  capabilities  which  significantly  affect  the  effectiveness  of 
the  development  environment. 


A-1 


TABLE  A-1 


Layer  3 Tool  List 


1.1  Requirements  Analysis 

1.1.1  General  Purpose  System  Simulators 

1.1.2  System  Description  Languages  and  Analyzers 

1.2  Software  Design 

1.2.1  Computer  System  Simulators 

1.2.2  Data  Base  Design  Aids 

1.2.3  Data  Dictionary  Systems 

1.3  Construct  System  Tests 

1.3.1  Test  Data  benerators 

1.3.2  Test  Data  Auditors 

1.3.3  Test  Case  Design  Advisors 

1.3.4  Test  I nstrumenters  and  Analyzers 

1 .4  Bui  1 d ana  Uni t T est 

1.4.1  Assemblers 

1 -H.  i . i Basic  Asser.ioler 

1.4. 1.2  i-iacro  Assembler 

1.4.2  Compilers 

1.4.3  Linkers 

1 .4.3.1  Basic  Linker 

1.4. 3. 2 Simple  Overlay  Linker 

1.4. 3. 3 Extended  Overlay  Linker 

1 .4.4  uebuggi ng  Ai ds 

1.4. 4.1  Interactive  Symbolic  Debugger 

1.4. 4. 2 Interactive  Absolute  Debugger 

1.4. 4. 3 Non-Interactive  Symbolic  Debugger 

1.4. 4. 4 Non-Interactive  Absolute  Debugger 

1.4.b  t'lodule  Libraries  in  Ciiange  Control  Systems 

1.4. b.l  Basic  or  Loosely  Coupled  Libraries 

1.4. b.2  Integrated  Libraries 

1.4. b.3  Automated  Software  Production  and  Test 


A-2 


1.4.6  Performance  Monitors 

1.4. 6.1  Language  Dependent  Monitors 

1.4. 6. 2 Language  Independent  Monitors 

1.4.7  Standards  Enforcers 


1.4.8  Preprocessors/Reformatters 

1.4. 8.1  Preprocessors 

1.4. 8. 2 Reformatters 


TABLE  k-i 


Layer  'i  Tool  List 


ZA  Data  Base  r-ianageinent  Systems 
Project  Managei.ient  Aias 

2.2.1  PERl/CPt'i  Systems 

2.2.<:  Project  Estimation  Systems 

2.3  Documentation  Aids 

2.3.1  Text  Processing  Systems 

2.3.2  Flowchart  Construction  Languages 

2.3.3  Automatic  Flowcharters 

2.4  Data  Manipulation  utilities 

2.4.1  Sort/Merge 

2.4.2  Editors 

2. 4. 2.1  Interactive  Source  Language  Editors 

2. 4. 2. 2 Interactive  Ooject  Module  Editors 
2.4.2.J  Batch  Source  Language  Editors 

2. 4. 2. 4  Batch  Ooject  Module  Editors 

2.b  Inforhiation  Retrieval  Systems 

2.b.l  Query  Languages 
2.b.2  Report  .iriters 


1 


TABLE  A-3 


Layer  1 List/Features 


3.1  Basic  Operating  Systems  (BOS) 

3.2  Multiprogramming  Operating  Systems  (MOS) 

3.3  Multiprocessor  Operating  Systems  (MPOS) 

3.4  Virtual  Machine  Monitors  (VMM) 

3.5  Time  Sharing  Operating  Systems  (TSOS) 

3.6  Real-Time  Operating  Systems  (RTOS) 


A. 1.1 


A. 1.1.1 


A. 1.1. 1.1 


Layer  3 Tools 

Tools  for  Requirements  Analysis 
General  Purpose  System  Simulator 


This  tool  allows  a user  to  construct  a computer  model  of  a real  or  pro- 
posed system  anci  to  perform  simulation  experiments  to  determine  the  behavior  of 
the  model  under  various  operatinq  conditions.  A "general  purpose"  simulator 
provides  the  modeler  with  very  basic  building  blocks  or  functions  that  can  oe 
parameterized  to  represent  computers,  magnetic  storage,  teleprocessing  lines, 
manual  operations,  etc.  These  functions  of  building  blocks  usually  have  names 
like  "queue,"  "server,"  "storage,"  “clock,"  and  so  on,  and  the  simulation  is 
controlled  by  the  occurrence  of  discrete  events.  A general  purpose  simulator 
must  provide  the  following  services; 

(1)  Allow  the  user  to  define  the  static  building  blocks  that  represent 
the  active  (processing)  and  passive  (queue  and  storage)  components  of  the  sys- 
tem. The  user  must  be  able  to  define  both  the  interconnection  of  system  elements 
and  their  capacities  or  processing  rates. 

(2)  Allow  the  user  to  specify  the  rate  at  which  transactions  are  generated 
to  drive  tne  system.  Common  arrival  distributions  including  random,  poisson, 
exponential  and  hyperexponential  should  be  available. 

(3)  Allow  tne  results  of  a siniulation  to  be  clearly  and  easily  displayed 
both  during  the  siinulation  and  at  its  end.  The  user  must  be  able  to  selectively 
display  critical  transaction  arrival  rates,  storage  utilizations,  processor 
utilizations,  and  queue  length  and  delay  statistics. 

A. 1.1. 1.2  System  description  Languages  and  Analyzers 

Assist  system  analysts  in  describing  the  functional  characteristics  of  a 
system,  and  in  validating  the  consistency  and  completeness  of  a functional  de- 
composition. The  System  Description  Language  should  allow  for  application  of  a 
top-down  methodology.  It  should  provide  language  facilities  to  define  both 
processes  or  activities  ano  data.  It  should  provide  facilities  to  quantify 
properties  of  both  processes  and  data.  Such  quantifiable  properties  may  be 
used  to  describe  performance,  size,  weight,  cost,  etc.  It  should  also  provide 
facilities  to  define  the  dynamic  behavior  of  the  system  in  sufficient  detail 
to  enable  the  generation  of  input  to  simulation  systems. 

A large  part  of  the  value  of  System  Description  Languages  and  Analyzers  is 
determined  by  the  types  of  reports  it  produces  to  assist  the  analysts  in  per- 
forming their  function,  communicating  with  other  analysts,  and  communicating 
with  users  and  management.  Representative  capabilities  that  should  be  present 
in  available  reports  are: 

(1)  Tne  ability  to  generate  a structured  description  of  any  process 
(i.e.,  constituent  subprocesses,  sub-subprocesses,  etc.) 

(2)  The  ability  to  generate  a structured  description  of  any  datum. 

(3)  The  ability  to  obtain  all  attributes  of  a process  or  datum. 
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(4)  The  ability  to  obtain  all  uses  of  a datum. 

(5)  The  ability  to  obtain  all  activation  conditions  of  a process. 

(6)  The  ability  to  obtain  all  processes  that  may  be  activated  by  an  event. 

(7)  The  ability  to  check  that  all  data  used  is  created  somewhere  within 

the  system  or  is  a system  input. 

(b)  The  ability  to  check  that  all  data  generated  within  the  system  is  used 
somewhere  or  is  a system  output. 

A. 1.1. 2 Tools  for  Software  Design 

A. 1.1. 2.1  Computer  System  Simulator 

This  tool  is  similar  in  nature  to  the  general  purpose  simulator,  except 
that  its  basic  building  blocks  represent  real  computer  system  components  whose 
modeled  oehavior  approximates  the  throughputs,  capacities,  and  access  times 
achievable  on  the  modeled  equipment.  The  model  should  contain  both  hardware 
characteristics  for  processor  speeds,  storage,  size,  access  time,  etc.,  but 
should  have  available  analytical  models  of  the  behavior  of  commonly  used  support 
software.  In  order  to  be  useful  to  the  software  designer,  the  computer  system 
simulator  should  possess  the  following  capabilities:  (1)  It  should  be  easy  to 
construct  a model  of  any  legally  configurable  group  of  hardware  components. 

(2)  It  should  be  possible  to  validate  the  results  of  a simulation  on  a subset 
of  the  available  hardware.  (3)  It  should  be  possible  to  alter  the  modeled  be- 
havior of  particular  components  to  correspond  to  different  software  algorithms. 
(4)  "Off-the-shelf"  models  for  common  storage  devices,  access  methods,  opera- 
ting systems,  and  so  on  should  be  available. 

A. 1.1. 2. 2 Data  Base  Design  Aids 

These  tools  assist  tne  data  base  designers  in  grouping  data  elements  into 
logical  record  classes  and  in  determining  the  relationships  among  logical  record 
classes  implicit  in  either  the  nature  of  the  data  or  the  usage  of  the  data.  The 
tool  should  also  provide  assistance  in  evaluating  the  performance  tradeoffs  among 
possible  physical  structures  for  a given  logical  data  base  structure  using,  as  a 
minimum,  frequency  of  usage,  size,  and  number  of  occurrences  data,  and  a variety 
of  access  methods  and  storage  placement  strategies.  The  tool  should  be  capable 
of  handling  one-to-one,  one-to-many,  and  many-to-many  relationships  among  logical 
record  classes. 

A. 1.1. 2. 3 Data  Dictionary  Systems 

These  tools  assist  data  base  designers  in  managing  the  data  definition 
activities.  During  succeeding  phases  of  development,  a data  dictionary  system 
provides  services  to  the  data  base  manager  to  assist  in  controlling  the  usage 
activities.  Data  dictionaries  define  data  elements  by  associating  with  the  data 
element  name  a set  of  properties  such  as  data  type,  external  representation, 
units,  value  set,  textual  data,  etc.  They  assist  in  controlling  the  usage  by 
identifying  logical  or  physical  record  classes  containing  the  data  element  and 
by  maintaining  the  history  of  changes  to  the  data  element  definition. 
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The  tool  typically  provides  a selective  retriv-val  capability.  Additional 
features  include  processors  to  generate  data  declarations  or  high  level  languages 
and  processors  to  generate  data  editing  and  validation  programs.  Some  COMPOOL 
facilities  can  be  classified  as  basic  data  dictionary  systems. 

A. 1.1. 3 Tools  for  the  Construction  of  System  Tests 

A. 1.1. 3.1  Test  Data  benerators 

Test  data  generators  create  data  files  for  testing  and  validating  computer 
programs.  The  test  data  file  may  oe  used  directly  as  input  to  the  module  under 
test  or  may  be  used  as  input  to  a driver  program  which  converts  the  test  data 
file  to  suitaole  stimuli  for  the  module  under  test.  Available  commercial  test 
data  generators  iiiay  be  directly  applicable  to  military  tactical,  and  command 
and  control  applications.  A test  data  generator  should  have  the  following 
capabi 1 i ties; 

(1)  Create  Mies  of  all  storage  organ! zations  supported  by  tne  host  opera- 
ting system,  i.e.,  sequential,  index  sequential,  random,  etc. 

(2)  Create  files  of  all  intra-file  structures  supported  by  the  host  opera- 
ting system,  i.e.,  fixed  and  variable  length  records,  hierarchical  record  organi- 
zations, etc. 

(3)  Accept  a suitable  description  of  data  elements  and  their  value  set. 

(4)  Accept  a suitaole  description  of  the  intrarecord  structure. 

(b)  Be  capable  of  generating  records  with  valid  and  invalid  (i.e..  witb’n 
or  outside  tlie  value  set)  data  elements  under  the  control  of  the  test  specifier. 

(b)  Provide  control  over  the  numbers  and  type  of  recoros  generated. 

(7)  Provide  control  over  the  logical  sequencing  of  records  within  the  file. 

(8)  Provide  control  over  the  physical  placement  of  records  within  the  file. 

A. 1.1. 3. 2 Test  Data  Auditors 

These  tools  compare  data  files  against  specifications  and  produce  reports 
of  discrepancies  and/or  compliance.  The  specifications  may  be  in  the  form  of 
another  data  file  or  in  the  form  of  file  and  record  descriptions  compatible 
with  a companion  Test  Data  Generator  tool.  This  tool  category  is  not  intended 
to  include  simple  file  comparison  programs  of  the  type  which  perform  a character 
by  character  or  word  by  word  comparison.  To  be  included  in  this  category  a Test 
Data  Auditor  should  have  the  following  capabilities. 

(1)  Accept  input  files  of  all  storage  organizations  supported  by  the  host 
operating  system,  i.e.,  sequential,  index  sequential,  random,  etc. 

(2)  Accept  input  files  of  all  intrafile  structures  supported  by  the  host 
pirating  system,  i.e.,  fixed  and  variable  length  records,  hierarchical  record 
' ia"' zations.  etc. 
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(3)  Accept  a suitable  description  of  data  elements  and  optionally  their 
value  set. 

(4)  Accept  a suitable  description  of  the  intrarecord  structure. 

(5)  Provide  control  over  the  number  and  type  of  records  audited. 

(6)  Provide  control  over  the  data  elements  to  be  audited  and  optionally 
over  the  type  of  comparison  to  be  performed. 

A. 1.1. 3. 3 Test  Case  Design  Advisors 

These  tools  analyze  programs  written  in  a high  level  language  and  present 
the  results  of  that  analysis  in  a form  suitable  to  assist  test  case  designers 
in  the  selection  of  test  data.  Suitability  of  presentation  of  analysis  results 
depends  on  the  test  data  selection  criterion.  There  is  no  known  test  data 
I selection  criterion  which  is  guaranteed  to  yield  a complete  set  of  tests,  i.e., 

f a set  of  tests  which,  if  successfully  passed,  assures  the  correctness  of  a pro- 

F gram.  However,  there  are  known  types  of  errors  which  will  not  be  detected  unless 

[ the  test  data  selection  criterion  satisfies  certain  conditions.  The  simplest 

such  condition  is  that  the  test  set  exercises  all  statements  in  a program. 

I Otherwise,  errors  present  in  the  unexercised  program  segments  could  not  be  found. 

[ A somewhat  more  comprehensive  criterion  is  to  exercise  all  branch  conditions  in 

I a program.  Such  criterion  will  not  only  exercise  all  statements  but  also  addi- 

[ tional  combinations  of  statement  sequences.  Another  useful  test  data  selection 

[ criterion  is  boundary  or  limit  test  data.  Such  test  cases  assist  in  verifying 

that  the  program  performs  correctly  within  the  valid  range  of  inputs  and  that  it 
adequately  protects  itself  from  invalid  data.  This  criterion  is  also  useful  in 
verifying  that  the  special  cases  typically  present  at  the  boundaries  are  adequately 
handlea. 

Test  case  design  analyzer  shoula  support  one  or  more  of  the  above  test  data 
selection  criteria.  As  a minimum  it  should: 

(1)  Identify  all  linear  program  segments,  i.e.,  all  sequences  of  state- 
ments such  that  if  Si  and  Sj  are  any  two  statements  in  the  sequence,  execution 
of  Si  implies  execution  of  Sj  and  vice  versa. 

(2)  Identify  predicates  that  must  be  satisfied  to  exercise  every  linear 
segment. 

(3)  Provide  assistance  in  tracing  the  predicate  variables  to  the  module's 
external  interfaces. 

(4)  Provide  assistance  in  identifying  valid  combinations  of  predicates, 
i.e.,  under  what  circumstances  a predicate  is  a false  tautology. 

A. 1.1. 3. 4 Test  Instrumenters  and  Analyzers 

These  tools  instrument  modules  under  test  so  as  to  collect  data  character- 
izing the  behavior  of  the  module.  The  collected  data  is  post-processed  by  an 
analyzer  which  produces  reports  to  aid  in  determining  the  success  of  the  test 
in  satisfying  a test  data  selection  criterion.  The  tool  should  support  one  or 
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more  of  the  test  data  selection  criteria  discussed  under  Test  Case  Design  Ad- 
visors. As  a minimum  it  should  identify  all  linear  segments  of  the  module  under 
test  not  covered  by  the  test. 

A. 1.1. 4 Tools  for  Building  * Unit-Testing  Software 
A. 1.1. 4.1  Assemblers 

Assemblers  allow  programs  to  be  codec  in  a symbolic  language  in  which 
statements  generally  correspond  to  a single  machine  instruction.  Allow  machine 
instructions  and  storage  areas  for  variable  and  constant  information  to  be  given 
; symbolic  labels,  and  permit  the  use  of  these  labels  in  assembler  statements. 

^ Permit  the  insertion  of  commentary  information  on  the  same  statement  as  a coded 

instruction  or  data  definition,  or  on  a separate  statement.  In  addition  to 
these  functions,  assemblers  may  have  other  capabilities  as  outlined  below. 

■ A. 1.1. 4. 1.1  Basic  Asseiiiolsr 

r 

I To  be  rmniinally  useful,  the  basic  assembler  must; 

(1)  Have  a minimum  label  size  of  b characters. 

t (2)  Allow  a label  to  be  used  in  a statement  before  it  has  oeen  defined. 

I (3)  Allow  the  explicit  declaration  of  external  symbols  and  entry  points. 

I (4)  Have  a symbolic  equivalent  for  the  value  of  tne  current  location 

i counter. 

: i 

(b)  Produce  a symbol  cross  reference  listing.  | 

(6)  I’roduce  error  diagnostics  that  identify  the  statement  ana  field  in 
error. 

(7)  Produce  a listing  with  both  symbolic  and  machine  language  instructions.  | 

A. 1.1. 3. 1.2  I'lacro  Assembler  | 

This  tool  provides  the  features  of  a basic  assembler,  plus  the  ability  to  i 

define  named  groups  of  multiple  basic  assembly  language  statements,  or  MACROS,  i 

that  can  be  inserted  in  line  in  an  assembly  language  program  simply  by  writing  ! 

the  name  of  the  macro.  In  addition  to  this  simple  code  substitution  capability,  j 

a macro  assembler  should  have  the  following  capabilities;  i 

i 

(1)  The  ability  to  modify  the  symbols  in  the  macro  instructions  by  sup-  • • j 

plying  keyword  or  positional  parameters  with  the  macro  when  it  is  invoked.  ' 

(2)  The  ability  to  modify  individual  fields  within  a single  instruction  . • j 

in  the  macro.  | 

(3)  The  ability  to  conditionally  generate  or  bypass  code  sequences  within  | 

the  macro  based  upon  values  passed  through  its  parameter  list. 
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(4)  The  ability  to  allow  the  macro  processor  to  check  the  validity  of  the 
parameter  list  coded  by  the  invoker  of  the  macro,  and  to  issue  warnings  when 
erroneous  invocations  occur. 

(5)  The  ability  to  invoke  an  inner  macro  within  the  definition  of  an 
outer  macro. 

(6)  The  ability  to  control  whether  or  not  the  expansion  of  a macro  is 
pri nted. 

A. 1.1. 4. 2 Compilers 

Compilers  translate  programs  written  in  a high  level  language  into  either 
relocatable  object  code  acceptable  to  a linker  or  assembly  language  acceptable 
to  an  assembler.  The  source  language  accepted  by  the  compiler  must  be  a subset 
or  dialect  approved  by  the  cognizant  DoD  agency.  Where  validation  standards 
have  been  established  (e.g.,  COBOL,  JOVIAL),  the  compiler  must  have  successfully 
passed  the  validation  test  as  interpreted  by  the  validation  agency.  The  require- 
ment of  relocatable  object  code  or  assembly  language  output  is  not  intended  to 
exclude  compilers  which  do  not  produce  directly  executable  code,  i.e.,  interpre- 
tive code.  Cross  compilers  for  an  architecture  other  than  the  CFA  under  evalua- 
tion are  excluded.  In  addition  to  the  primary  object  code  output,  the  compiler 
should  provide  the  following  capabilities  preferably  under  option  control; 

(1)  Source  listing 

(2)  Symbol  dictionary  listing  incluoing  symbol  attributes  | 

(3)  Cross  reference  listing 

(4)  Suitable  error  diagnostics 

(b)  Debugging  aids  support 

A. 1.1. 4, 3 Linkers 

Linkers  combine  the  text  produced  by  separate  invocations  of  compilers  and 
assemblers  ("object  modules")  into  executable  code  strings  ("load  modules"  or 
"core  images")  that  can  be  loaded  into  the  computers  main  storage  and  executed 
without  further  pre-processing. 

A. 1.1. 4. 3.1  Basic  Linker 

The  basic  linker  must  perform  at  least  the  following  functions: 

(1)  Allow  the  user  to  supply  control  statements  that  can  control: 

(a)  the  absolute  main  storage  address  that  is  to  be  the  origin  of 
the  resulting  load  module 

(b)  the  sequence  in  which  the  input  object  decks  are  to  be  loaded 

(c)  the  symbolic  name  of  the  resulting  load  module 
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(2)  Resolve  all  external  symbol  references. 

(3)  Relocate  all  required  addresses  that  were  specified  relative  to  the 
load  point. 

(4)  Produce  error  diagnostics  that  list  any  unresolved  external  symbols. 

(b)  Produce  a load  module  map  that  shows  the  absolute  address  of  all  ex- 
ternal symbols,  entry  points,  and  the  boundaries  of  the  consistent  object 
modules. 

(6)  Produce  a cross  reference  listing  for  all  external  references  and 
entry  points,  including  external  symbols  that  are  unreferenced. 

A. 1.1. 3. 3.1  Simple  Overlay  Linker 

In  addition  to  the  services  of  the  basic  linker,  the  simple  overlay  linker 
allows  the  user  to  define  phases  or  overlays  of  load  module  code  in  which  certain 
object  modules  are  linked  into  the  same  physical  main  storage  locations.  In  the 
case  of  the  simple  overlay  linker,  only  the  resolution  of  external  symbol  ref- 
erences and  the  relocation  of  address  references  is  handled;  controlling  the 
contents  of  storage  at  execution  time  is  the  responsibility  of  the  user  program. 
The  simple  overlay  linker  provides  the  following  additional  services: 

(1)  Ability  to  define  a "root  phase,"  or  a group  of  object  modules  that 
are  linked  into  a load  module  that  is  alwaysr  resident  in  main  storage  and  is 
never  overlayed  by  other  code. 

id)  Ability  to  specify  which  object  inooules  are  to  be  grouped  into  "over- 
lay phases"  that  may  be  read  into  the  same  area  of  storage  at  the  direction  of 
the  root  phase  usually  via  calls  to  supervisor  functions. 

(3)  Ability  to  specify  a simple  tree  of  segments  of  overlays. 

A. 1.1. 4. 3. 2 Extended  Overlay  Linker 

The  major  extensions  this  tool  has  over  the  simple  overlay  linker  is  the 
addition  of  the  following  capabilities: 

(1)  The  ability  to  assist  the  user  with  storage  cotitent  management  oy 
automatically  detecting  and  routing  inter-segment  calls  or  transfers  of  control 
through  the  supervisor  in  order  to  invoke  the  loading  of  the  required  phases. 

(2)  The  ability  to  define  more  complex  overlay  trees  that  may  consist  of 
several  discontiguous  regions. 

(3)  The  ability  to  "edit"  object  and  load  modules  by  rearranging,  re- 
placing or  deleting  their  constituent  object  modules. 

(4)  The  ability  to  perform  automatic  module  library  searches  in  order  to 
resolve  external  references. 
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I (5)  The  ability  to  record  extensive  information  about  the  characteristics 

!of  the  load  module  (e.g.,  whether  or  not  it  is  reentrant,  whether  it  is  exe- 
cutable, etc.). 

A. 1.1. 4. 4 Debugging  Aids 

[ These  tools  assist  the  programmer  in  locating  the  sources  of  program  errors 

I that  have  been  discovered  during  unit  testing,  usually  by  giving  him  some  control 

; over  the  execution  of  the  module  under  test  that  is  external  to  the  normal  pro- 

gram code.  Debugging  aids  can  be  classed  as  symbolic  or  absolute  and  as  inter- 
active and  non-interactive.  Symbolic  aids  allow  the  programmer  to  manipulate 
the  instructions  and  data  in  the  load  module  under  test  by  using  the  symbolic 
labels  and  data  names  in  the  source  program  (assembler  or  HOL)  that  were  used  to 
create  the  load  module.  Symbolic  debugging  aids  are  always  language  dependent 
and  they  require  that  the  compiler  or  assembler  produce  a machine  readable  symbol 
table.  Absolute  debugging  aids  require  that  the  programmer  reference  his  module 
by  using  absolute  addresses;  however,  they  may  be  used  in  cases  where  the  source 

‘ language  translator  does  not  supply  a symbol  table.  Interactive  debugging  aids 

are  designed  for  use  in  a time  sharing  environment.  They  allow  the  programmer 
to  debug  his  program  as  if  he  were  controlling  the  computer  through  the  switches 
on  a control  panel.  Non-interactive  debugging  aids  require  that  debugging  con- 
trol be  enforced  via  control  cards  in  the  batch  job  stream.  Minimum  requirements 
for  each  type  of  debugging  aid  are  contained  in  the  following  section. 

A. 1.1. 4. 4.1  Interactive  Symbolic  Debuggers 

(1)  Must  give  control  to  the  user  prior  to  initiation  of  the  module  under 

test, 

(2)  Must  allow  all  labeled  data  and  instructions  to  be  displayed  and  altered, 

(3)  Must  allow  the  insertion  of  breakpoints, 

(4)  Must  have  a single  step  execution  mode, 

(6)  Must  allow  tl.e  program  to  be  restarted  after  a breakpoint  either  at  the 
next  instruction  or  at  a different  location, 

(6)  Must  allow  the  user  to  obtain  on-line  or  off-line  snapshot  or  full 
storage  dumps, 

(7)  Must  allow  the  display  and  alteration  of  machine  facilities  (as  opposed 
to  symbolic  locations  and  variables)  such  as  status  words,  index  registers,  etc. 

One  Iterative  Symbolic  Debugger  should  be  evaluated  for  each  source  language. 

A. 1.1. 4. 4. 2 Interactive  Absolute  Debugger 

Requirements  for  this  tool  are  the  same  as  for  the  symbolic  debugger,  except 
that  references  to  program  and  data  storage  are  by  absolute  or  relative  main 
storage  address. 
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A. 1.1. 4. 4. 3 Non-Interactive  Symbolic  Debugger 

Because  the  program  under  test  will  not  be  under  the  direct  control  of  a 
programmer  when  this  tool  is  in  use,  it  must  have  more  extensive  options  than 
the  interactive  version.  In  addition  to  all  the  display,  alteration  and  restart 
capabilities  mentioned  above,  the  non-interactive  symbolic  debugger  must: 

(1)  Allow  the  setup  of  calling  parameter  lists,  global  variables,  and  so 
on  prior  co  module  execution, 

(2)  Allow  the  selective  invocation  of  instruction  traces  for  all  or  parts 
of  a program.  In  order  to  restrict  the  volume  of  output,  the  amount  of  data  dis- 
played should  be  optional,  and  the  invocation/termination  of  the  trace  should  be 
controllable  based  on  loop  counts  or  the  value  of  variables  in  the  program  under 
test. 

A. 1.1. 4. 4. 4 Non-I nteracti ve  Absolute  Debugger 

See  paragraphs  A. 1.1. 4. 4. 2 and  A. 1.1. 4. 4. 3 above. 

A. 1.1. 4. 5 Module  Libraries  & Change  Control  Systems 

These  tools  provide  computer  controlled  maintenance  of  groups  of  related  ! 

source  modules  (programs),  object  modules  (the  output  of  assemblers  and  compilers), 
and  load  modules  (the  output  of  linkers).  Depending  upon  their  sophistication,  , 

they  may  also  enforce  various  consistency  checks,  as  described  below.  i 

A. 1.1. 4. 5.1  basic,  or  Loosely  Coupled  Libraries  ^ 

These  are  the  lowest  level,  least  integrated  libraries;  they  are  simply  col- 
lections of  named  modules.  These  libraries,  to  oe  at  all  useful,  must  have  the 
following  attrioutes; 

(1)  Their  constituent  modules  must  be  accessible  by  name  by  the  appropriate 

translators.  For  example,  a compiler  must  be  able  to  read  a source  module  direct- 
ly from  a source  library  and  place  its  output  directly  in  an  object  module  library, 
without  the  intervention  of  data  manipulation  utilities.  ' 

(2)  Utility  programs  that  can  convert  modules  in  a library  to  other  media 
must  be  available,  as  well  as  utilities  to  reclaim  unused  library  space. 

(3)  The  library  management  software  must  prevent  the  construction  of  mod- 
ules with  duplicate  names,  and  must  have  safeguards  against  inadvertent  module  ; 

destruction.  (For  example,  replacement  of  an  "old"  module  by  a "new"  version  , 

with  the  same  name  should  require  an  explicit  replacement  request,  rather  than  ’ i 

be  the  default  when  a duplicate-name  module  is  added  to  a library.)  i 

j 

(4)  Basic  programs  to  display  the  contents  (module  names)  and  unused  • j 

space  of  a library  must  exist.  I 


A-14 


A. 1.1. 4. 5, 2 Integrated  Libraries 


Integrated  libraries  torni  a system  of  oasic  libraries  whose  module  members 
are  related  and  among  which  consistency  is  enforced.  For  example,  the  enforce- 
ment software  should  not  allow  a source  module  to  be  updated,  edited  or  replaced 
unless  an  entry  in  the  ECO  (Engineering  Change  Order)  Library  exists  that  names 
the  affected  module.  Furthermore,  original  modules  are  never  replaced  unless 
they  are  first  archived  to  a long  term  storage  medium.  Minimum  requirements  of 
an  integrated  library  system  are  as  follows: 

(1)  Internal  consistency  between  source  modules,  object  modules,  documenta- 
tion, and  test  case  streams  must  be  enforced  automatically. 

(2)  Hard  copy  auait  listings  must  be  produced  for  all  changes. 

(3)  Archiving  of  old  modules  should  be  automatic  and  fail  safe. 

A.1.1.4.b.3  Automated  Software  Production  and  Test  Systems 

These  tools  add  the  following  capabilities  to  integrated  libraries: 

(1)  Inter-library  and  inter-module  cross  reference  tables  are  automatically 
maintained.  For  example,  in  an  integrated  library  'ysten,  a p'^ngrarc.c’'  still  must 
manually  determine  all  the  affected  modules  to  specify  in  an  ECO  that  requires  a 
change  to  a system  subroutine.  In  an  automated  production  and  test  system,  the 
automatically  produced  and  maintained  cross  reference  tables  will  immediately 

sho-  which  tuuCulos  in.-  :.odu1e  to  be  changc-o. 

(2)  Furthermore,  the  inter-library  links  to  the  regression  test  library 
will  indicate  which  tests  niust  be  run  to  verity  that  the  change  has  had  no  un- 
wanted side  effects. 

(3)  The  test  case  resu’t  library  will  indicate  which  result  files  will 
have  to  be  altered  or  extended. 

(4)  The  consistency  checking  software  will  then  guarantee  that  a new  sys- 
tem load  module  is  not  released  until  all  required  modifications  to  source  code, 
documentation  and  test  cases  have  been  made;  all  compilations  and  linker  runs 
executed;  and  all  required  regression  tests  successfully  passed. 

A. 1.1. 4. b Performance  Moni tors 

These  tools  assist  the  programmer  in  quantifying  the  resource  consumption 
characteristics  of  a program,  and  in  isolating  performance  critical  areas.  Per- 
formance monitors  may  oe  classified  as  either  dependent  on  or  independent  from 
source  language. 

A. 1.1. 4. 6.1  Language-Dependent  Monitors 

These  tools  allow  the  programmer  to  obtain  "fine-grained"  statistics  on 
program  execution  (below  the  level  of  the  program  or  subroutine)  by  inserting 
statements  that  instrument  the  program  under  test  in  order  to  obtain  information 
such  as  the  number  times  critical  loops  are  executed,  average  depth  of  queue  and 
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list  searches,  numbers  of  calls  to  certain  subroutines,  and  so  on.  These  moni- 
tors are  language  dependent  in  the  sense  that  they  allow  the  programmer  to  in- 
sert "probes"  into  the  source  language  program  that  must  be  compiled  into  calls 
to  the  monitor  subroutines.  Language-dependent  monitors  must  provide  the  fol- 
lowing functions; 

(1)  The  ability  to  count  event  occurrences  defined  by  the  programmer, 
such  as  entries  to  subroutines,  loop  iterations,  etc. 

(i^)  The  ability  to  obtain  subroutine  or  loop  timings.  The  timings  must 
allow  for  the  perturbations  introduced  by  the  monitor  probe  itself. 

(3)  The  ability  to  maintain  averaging  counters  by  event  and  integrating 
counters  over  time. 

(4)  The  ability  to  summarize  a module's  execution  characteristics  by  per- 
cent of  time  spent  in  each  program  segiiient  (wHILE  or  FUR  loop,  IF-THEM-ELSE  leg, 
straiglit  segment,  closed  subroutine)  for  a given  set  of  test  data. 

A.1.1.4.b.i;  Language-Independent  '-lonitors 

These  tools  allow  the  programmer  to  measure  a module's  performance  charac- 
teristics in  terms  of  its  interactions  with  the  other  components  of  the  system 
(operating  systems,  library  routines,  other  user  programs)  with  which  it  executes. 
This  information  must  usually  be  gathered  by  activating  probes  that  monitor  the 
program's  invocation  of  such  operating  system  services  as  I/O  requests,  overlay 
requests,  timer  requests,  and  external  module  calls.  Language  independent  moni- 
tors may  also  make  use  of  CPU  utilization  figures  maintained  by  the  operating 
system  for  accounting  or  resource  management  purposes  and  on  virtual  and  real 
memory  utilization  matters.  They  may  also  take  advantage  of  the  existence  of 
special  event  recording  hardware  in  the  host  processor,  usually  in  conjunction 
with  some  degree  of  operating  system  support.  Language-i noepenuent  monitors 
should  have  the  following  capabilities: 

(1)  The  ability  to  record  the  details  of  a module's  utilization  of  system 
services  such  as  I/O  requests,  timer  service  requests,  and  external  calls  that 
must  be  mediatea  by  the  supervisor, 

(2)  The  ability  to  record  typical  execution  versus  wait  time  patterns 
that  a program  exhibits. 

(3)  The  ability  to  record  real  storage  utilization  patterns. 

(4)  For  demand  paged  virtual  memory  systems,  the  ability  to  monitor  and 
record  a module's  instruction  and  data  reference  patterns  over  time,  along  with 
the  ability  to  record  its  influence  on  paging  rates. 

A. 1.1. 4. 7 Stanoards  Enforcers 

These  tools  allow  source  programs  to  be  automatically  examined,  and  checked 
for  conformance  to  installation-defined  standards  of  format,  content,  and  usage. 
Since  they  must  be  able  to  recognize  source  language  constructs,  they  are  lan- 
guage-dependent. Standards  enforcers  should  be  able  to  perform  at  least  the 
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following  functions,  and  the  requirements  that  are  to  be  enforced  should  be  eusiij 
parameterized  by  the  using  installation: 

(1)  Check  for  binary  conditions  such  as  the  presence  of  unwanted  language 
constructs,  or  the  absence  of  explicit  data  type  declarations  even  when  these 
conditions  are  permitted  by  the  language. 

(2)  Check  for  quantitative  conditions  such  as  number  of  programs  statements 
per  module,  ratio  of  comments  to  computational  statements,  or  number  of  parameters 
per  suorouti ne  call. 

(3)  Allow  the  installation  to  define  qualitative  measures  such  as  expression 
complexi ty . 


A. 1 .1 .4.8  Preprocessors/Reformatters 

These  tools  assist  programmers  in  producing  well-structured  and  readable 
programs  by  allowing  the  introduction  of  wel 1 -structured  programming  constructs 
into  source  program  for  languages  that  do  not  have  them  (e.g.,  structured  pro- 
gramming preprocessors  for  FORTRAN),  and  by  automatically  controlling  the  inden- 
tation of  nested  IF-THEN-ELSE  clauses  and  loops,  the  placement  of  comments,  and 
so  on  to  produce  readable  listings.  The  use  of  these  tools  not  only  assists  the 
production  of  more  maintainable  programs,  but  also  increases  programmer  produc- 
tivity by  freeing  the  programmer  from  the  need  to  observe  strict  columnization 
rules  when  preparing  source  module  for  input. 

A. 1.1. 4. 8.1  Preprocessors 

Preprocessors  should  allow  the  definition  of  a simple  structured  program- 
ming constructs  that  produce  suitable  statements  in  the  target  source  language. 
These  should  incluoe  the  IF-THEN-ELSE,  UO-WHILE,  and  CASE  constructs  at  a minimum. 

A. 1.1. 4. 8. 2 Reformatters 

Reformatters  should  allow  the  using  installation  to  flexibly  define  para- 
meters for  loop  and  IF-THEN-ELSE  indentation,  comment  placement,  and  page  ejec- 
tion control.  The  reformatter  should  allow  the  source  language  programmer  to 
make  maximum  use  of  printer  lines  that  are  longer  than  the  image  of  the  input 
statements. 

A. 1.2  Layer  2 Tools 

A.  1.2.1  Data  Base  Management  Systems 

These  systems  allow  the  user  of  a computer  system  to  define  the  contents 
and  the  logical  relationships  between  collections  of  data  items  that  represent 
some  useful  abstraction  of  a real-world  phenomenon  (command  and  control  func- 
tions, the  modules  and  documentation  of  a system  of  computer  programs)  without 
being  concerned  with  the  physical  mechanics  of  storing,  locating  and  retrieving 
items  or  groups  of  items.  While  data  base  management  systems  have  usually  been 
designed  to  meet  the  needs  of  the  commercial  data  processing  market,  they  can 
provide  a suitable  base  for  a large,  integrated  software  production  facility 
and  provide  the  framework  for  a large  military  system.  In  order  to  be  a useful 
candidate  for  this  role,  a data  base  management  system  would  have  to  possess 
the  following  characteristics; 


A-17 


i 

t 


f 

i 


I 


(1)  It  should  have  the  capability  to  store  logically  equivalent  elements 
of  varying  size.  It  does  not  appear  useful  to  attempt  to  manage  program  mod- 
ules at  the  source  statement  level;  therefore,  the  system  must  be  able  to  store 
and  maintain  the  logical  relationships  between  varying  sized  units  of  source 
programs,  documents,  test  case  files,  test  result  files,  and  so  on. 

(2)  It  must  support  dynamically  varying  structures,  as  different  user 
requirements  create  the  need  to  define  different  inter-unit  relationships, 
different  inter-mooule  dependencies,  etc. 

(3)  It  should  provide  a suitable  interface  to  the  programs  that  must 
extract,  modify  and  add  data  to  the  system,  including  language  translators, 
linkers,  editors,  t ocumentation  aids,  report  writers,  and  project  management 
aids. 


(4)  It  should  handle  the  problems  of  transaction  journaling  archiving, 
back  up  and  recovery  in  such  a way  that  the  physical  integrity  of  the  data  base 
is  guaranteed. 

(5)  It  must  permit  the  logical  consistency  of  the  data  to  be  easily  main- 
tained oy  user  written  utilities. 

(6)  It  must  have  an  effective  security  system  that  will  prevent  inadver- 
tent or  unauthorized  access  or  destruction  of  data. 

A. 1.2. 2 Project  Management  Aids 

A.  1.2. 2.1  PERT/CPM  Systems 

PERT/CPM  Systems  assist  managers  in  planning  and  controlling  project  activ- 
ities. As  a minimum,  the  system  should  provide  two  types  of  project  elements 
(e.g.,  phases  and  tasks)  and  provide  the  following  capabilities: 

(1)  Specification  of  personnel  and  equipment  resource  requirements  at  the 
task  level . 

(2)  Specification  of  actual  resource  consumption  at  the  task  level. 

(3)  Project  scheduling  reports  identifying  potential  slippages,  earliest 
and  latest  initiation/completion  times  of  tasks,  and  potential  trouble  areas. 

(4)  Project  staffing  reports  identifying  required  and  possible  staffing 
level s. 

(5) '  Equipment  usage  reports  identifying  required  and  possible  needs  cor- 
related with  staffing  projections. 

(6)  Project  status  reports  identifying  accomplishments  in  relation  to  plan. 
A. 1.2. 2. 2 Project  Estimation  Systems 

These  systems  assist  in  the  develoment  of  work  breakdown  structures  and 
related  performance  standards  for  use  in  estimating  project  resource  requirements. 
The  tools  should  have  the  following  capabilities: 
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(1)  Support  multi-level  work  breakdown  structures. 

(2)  Collection  of  performance  data  for  multiple  instances  of  a work  break-  | 

down  structure.  j 

(3)  Develop  and  update  performance  standards  from  collected  data.  | 

I 

(4)  Compute  statistical  characteristics  of  the  collected  data  to  assist  j 

in  evaluating  the  i^eli ability  of  the  standards.  I 

(b)  Provide  reports  to  assist  in  identifying  reasons  for  significant  devia- 
tion of  actual  performance  from  the  standard. 

A. 1.2. 3 Documentation  Aids 

These  tools  assist  the  user  in  the  preparation  and  maintenance  of  documenta- 
tion about  the  modules  of  a system.  Aids  most  relevant  to  a program  development 
environment  are  as  follows.  ] 

A. 1.2. 3.1  Text  Processing  Systems 

These  systems  allow  the  maintenance  of  printed  documents  in  machine  readable  1 

form.  In  addition  to  the  actual  printed  text  of  the  document,  the  machine  read- 
able file  generally  contains  control  statements  that  specify  the  format  of  the 
finishea  document  in  terms  of  spacing,  indentation,  pagination,  titling,  and 
justification.  More  advanced  systems  may  allow  documents  to  be  automatically 
constructed  from  fragments  that  can  be  dynamically  combined  at  execution  time, 
and  can  automatically  construct  indices  and  tables  of  contents.  These  systems 
must  be  combined  with  an  online  text  editor  to  be  useful. 

A. 1.2. 3. 2 Flowchart  Construction  Languages 

These  tools  allow  the  construction  of  programs  that  can  be  compiled  into 
printed  or  plotted  flowcharts.  The  statements  used  to  specify  the  flowcharts 
may  usually  either  be  kept  in  separate  files  or  be  embedded  as  comments  in  the 
source  files  of  the  programs  whose  logic  flow  they  describe.  This  type  of 
automated  flowcharting  has  the  advantage  of  allowing  the  programmer  to  specify 
the  processing  and  decision  blocks  of  the  chart  at  an  appropriate  level  of  de- 
tail. It  has  the  disadvantage  of  creating  yet  another  "program"  that  must  be 
maintained  and  that  is  as  likely  as  not  to  become  inconsistent  with  its  associ- 
ated source  program. 

A. 1.2. 3. 3 Automatic  Flowcharters 

Automatic  flowcharters  use  the  source  language  programs  themselves  as  in- 
put to  construct  a flowchart.  This  method  has  the  obvious  advantage  of  always 
being  able  to  obtain  a flowchart  that  is  an  accurate  representation  of  the 
source  module.  It  has  the  less  obvious  disadvantage  of  usually  producing 
flowcharts  that  are  specified  at  a greater  level  of  detail  than  is  desirable. 

There  exists,  of  course,  an  overriding  question  of  the  utility  of  flowcharts 
as  system  documentation  at  any  but  the  highest  system  block  diagram  level; 
many  people  feel  that  a wel 1 -documented  structured  program  provides  a more 
suitable  reference. 
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A. 1.2. 4 


Data  Manipulation  Utilities 


These  tools  allow  the  system  user  to  alter  the  format  and  content  of  data 
files  inaependent  of  the  logical  significance  of  the  data  fields  involved. 

While  this  category  includes  the  standard  media  conversion  utilities  such  as 
card-to-tape,  tape-to-pri nter,  etc.,  these  have  not  oeen  included  because  it 
is  felt  that  simple  programs  of  this  type  are  almost  universally  available. 

The  evaluation  of  tools  in  this  category  will  therefore  be  restricted  to  sort/ 
merge  programs  and  editors. 

A.  1.2. 4.1  Sort/Merge 

This  utility  allows  the  user  to  rearrange  the  order  of  the  logical  records 
in  one  or  more  input  files  so  that  the  records  of  the  resulting  output  file 
are  in  oraer  specified  by  the  collating  sequence  of  a series  of  one  or  more 
fields  in  each  record.  A usable  sort/merge  program  should  possess  the  fol- 
1 owing  capabi 1 i ties: 

(1)  It  should  process  multiple  input  files. 

(2)  It  should  permit  sorting  and  merging  on  multiple  fields. 

(3)  It  should  allow  a mixture  of  ascending  and  descending  sequences  to 
be  specified  for  different  fields  in  the  same  run. 

(4)  It  should  allow  specification  of  any  meaningful  data  type  for  a 
sorted  field  (e.g.,  binary,  ASCII,  floating  point,  etc.). 

(b)  It  should  allow  use  of  any  available  checkpoint/restart  facilities. 

(b)  It  should  permit  exits  to  be  taken  to  user  programs  at  appropriate 
points  (e.g.,  for  end  of  volume  processing,  or  to  edit  and  delete  individual 
input  or  output  records). 

A. 1.2. 4. 2 Editors 

Editors  are  programs  that  allow  the  user  to  ado,  delete,  replace  and  alter 
the  contents  of  individual  data  records  within  a file.  They  may  be  divided 
into  interactive  editors  that  are  designed  to  be  used  in  an  on  line,  time- 
sharing mode,  and  batch  editors  that  take  their  commands  from  control  cards  in 
the  input  stream.  Both  types  of  editors  may  be  further  subdivided  into  those 
that  are  primarily  intended  for  use  in  developing  source  programs,  and  those 
that  are  designed  to  alter  or  "patch"  binary  object  or  load  modules.  The  re- 
quired features  of  each  type  of  editor  are  summarized  below. 

A. 1.2. 4. 2.1  Interactive  Source  Language  Editors 

These  editors  allow  the  user  at  a CRT  or  typewriter  terminal  to  examine 
and  alter  source  language  statements.  There  exists  a wide  variety  of  editors 
in  this  category,  and  the  features  cited  below  are  intended  to  represent  the 
minimum  functional  requirements  of  a useful  source  editor: 

(1)  Must  be  able  to  locate  items  both  by  content  (string  comparison)  and 
line  number. 
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iZ)  Must  be  able  to  restrict  the  field  of  string  search  (and  alteration) 
to  be  within  fixed  columns  (to  avoid  altering  sequence  or  statement  numbers). 

(3)  Must  be  able  to  make  global  (file  wide)  changes  of  one  string  to 
another. 

(4)  Default  actions  and  abbreviations  should  be  fail-safe.  For  example, 
one  well-known  editor  interprets  a carriage  return  after  a null  line  as  a re- 
quest to  delete  the  data  record  last  displayed.  This  is  a very  poor  practice. 

(5)  Must  be  possible  to  save  and  retrieve  intermediate  edit  sessions  in 
case  of  system  failure. 

A. 1.2. 4. 2. 2 Interactive  Object  Module  Editor 

This  editor  can  be  used  to  alter  binary  object  or  load  modules  by  inserting 
patches.  Its  purpose  can  be  served  oy  the  source  editor  if  that  program  has  an 
option  that  allows  a file's  contents  to  be  displayed  and  altered  in  hexadecimal 
or  octal  from  a terminal. 

A. 1.2. 4. 1.3  Batch  Source  Language  Editors 

These  editors  usually  work  on  a line  number  basis,  using  information  sup- 
plied in  control  cards  to  add,  delete  or  replace  records  in  a particular  file. 
The  usability  and  human  factors  suitability  of  the  programs  vary  widely,  but, 
as  a minimum,  a batch  editor  must: 

(1)  Allow  insertion,  replacement,  deletion  and  renumbering  of  the  source 

file. 

(2)  Print  a log  of  its  actions  that  shows  all  alterations  to  the  file, 
and  provides  enough  line  number  information  to  allow  future  changes  without  ob- 
taining a complete  new  listing. 

A. 1.2. 4. 1.4  Batch  Object  Module  Editors 

These  binary  patching  programs  should  work  either  on  a card  sequence  num- 
ber (if  it  exists  in  the  object  library)  or  on  a relative  byte  or  word  address 
within  the  module.  Furthermore,  the  editor  should  have  the  ability  to  verify 
that  the  current  contents  of  the  locations  to  be  changed  match  a hexadecimal 
or  octal  string  supplied  by  the  user  prior  to  altering  the  locations  to  its 
new  value.  A change  log  should  also  be  produced. 

A.l.2.b  Information  Retrieval  Systems 

These  systems  are  general  purpose  application  programs  operating  either 
on-line  (interactively)  or  in  the  batch  that  interpret  user  requests  tc  locate 
and  display  information  that  is  stored  either  within  a structured  data  base  or 
within  separate  files.  These  systems  can  be  classified  either  as  query  language 
systems  or  as  report  writers. 


